903: LLM Benchmarks Are Lying to You (And What to Do Instead), with Sinan Ozdemir
Super Data Science: ML & AI Podcast with Jon Krohn
00:00
Challenging the Validity of Language Model Benchmarks
This chapter critiques traditional benchmarking methods for language models, highlighting biases in multiple-choice questions. It advocates for domain-specific evaluations, such as the Software Engineering Benchmark, and explores ways to improve consistency in assessing AI capabilities.
Transcript
Play full episode
