Nandan Thakur

Nandan Thakur, a Ph.D. student at the University of Waterloo, dives deep into Retrieval-Augmented Generation (RAG) and its significant benchmarks like BEIR and MIRACLE. He discusses the evolution of embedding models and the balance between specialization and generalization. The conversation highlights advancements in query decomposition, emphasizing new methods for complex user queries. Nandan also explores the complexities of summarizing AI search results and the importance of nuanced evaluations in RAG benchmarks for real-world applications.

Nandan Thakur, the first author of the BEIR benchmark paper, dives deep into the realm of Information Retrieval. He discusses the current state of model evaluations, shedding light on the uphill battle of models trying to surpass the BM25 baseline. Nandan highlights the shortcomings of BEIR as of 2024 and addresses specific challenges like the Touché 2020 subset. He also shares insights on the future of benchmarking, including the newly announced TREC RAG track, and emphasizes the critical role of multilingual capabilities in advancing IR systems.