2min snip

Vanishing Gradients cover image

Episode 21: Deploying LLMs in Production: Lessons Learned

Vanishing Gradients

NOTE

Importance of Benchmarks Correlating with Reality in Model Evaluation

Offline evaluation metrics on generic tasks for applied problems may not be trustworthy due to potential noise and issues such as data leakage. It is essential to find benchmarks that correlate with real-world experiences to evaluate models effectively. By identifying benchmarks that align with one's reality after experimenting with different models, one can increase trust in the evaluation process and make more informed decisions.

00:00

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode