In this enlightening seventh episode of Practically Intelligent, we take a look at the pivotal role of benchmarking in advancing AI with Praveen Paritosh, a leading figure in AI research. Discover why shared benchmarks are not just important, but critical in pushing the boundaries of AI technology. Praveen enlightens us on the legacy benchmarks like SQuAD, instrumental in testing early question-answer systems, and how they paved the way for early leaderboards in AI. We discuss the concept of shared benchmarks as a mechanism for the research community to collectively tackle and progress in specific challenges, drawing parallels between NLP and image recognition benchmarks like ImageNet. However, it's not all straightforward – benchmarks, while guiding us in the right direction, are merely proxies. We discuss the challenges of differentiating between conceptual learning driven by reasoning and rote learning based on memorization. Join us for a deep dive into the intricacies and nuances of AI benchmarking, a critical yet complex tool in the evolution of artificial intelligence.