Evaluating Language Models
The Data Exchange with Ben Lorica
00:00
How to Measure Efficiency in the API
On efficiency, let's talk about inference efficiency. How long does it take for me to get a result back? And this is a useful thing to know because if you want to use the API, you kind of want to know. It depends on the hardware they have back. So there's a model efficiency where it's only probably a model. There's optimizations like caching and batching and various tricks. And then there's hardware. Like if you add more GPUs, sure, in many cases you can get things to go faster.
Transcript
Play full episode
