Evaluating Extrapolation Performance of Dense Retrieval: How does DR compare to cross encoders when it comes to generalization?

How much of the training and test sets in TREC or MS Marco overlap? Can we evaluate on different splits of the data to isolate the extrapolation performance?

In this episode of Neural Information Retrieval Talks, Andrew Yates and Sergi Castella i Sapé discuss the paper "Evaluating Extrapolation Performance of Dense Retrieval" byJingtao Zhan, Xiaohui Xie, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma.

📄 Paper: https://arxiv.org/abs/2204.11447

❓ About MS Marco: https://microsoft.github.io/msmarco/

❓About TREC: https://trec.nist.gov/

🪃 Feedback form: https://scastella.typeform.com/to/rg7a5GfJ  

Timestamps: 

00:00 Introduction 

01:08 Evaluation in Information Retrieval, why is it exciting 

07:40 Extrapolation Performance in Dense Retrieval 

10:30 Learning in High Dimension Always Amounts to Extrapolation 

11:40 3 Research questions 

16:18 Defining Train-Test label overlap: entity and query intent overlap 

21:00 Train-test Overlap in existing benchmarks TREC 

23:29 Resampling evaluation methods: constructing distinct train-test sets 

25:37 Baselines and results: ColBERT, SPLADE

29:36 Table 6: interpolation vs. extrapolation performance in TREC 

33:06 Table 7: interplation vs. extrapolation in MS Marco 

35:55 Table 8: Comparing different DR training approaches 

40:00 Research Question 1 resolved: cross encoders are more robust than dense retrieval in extrapolation 

42:00 Extrapolation and Domain Transfer: BEIR benchmark. 

44:46 Figure 2: correlation between extrapolation performance and domain transfer performance 

48:35 Broad strokes takeaways from this work 

52:30 Is there any intuition behind the results where Dense Retrieval generalizes worse than Cross Encoders? 

56:14 Will this have an impact on the IR benchmarking culture? 

57:40 Outro   

Contact: castella@zeta-alpha.com