ColBERT + ColBERTv2: late interaction at a reasonable inference cost

Andrew Yates (Assistant Professor at the University of Amsterdam) and Sergi Castella (Analyst at Zeta Alpha) discus the two influential papers introducing ColBERT (from 2020) and ColBERT v2 (from 2022), which mainly propose a fast late interaction operation to achieve a performance close to full cross-encoders but at a more manageable computational cost at inference; along with many other optimizations.

📄 ColBERT: "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT" by Omar Khattab and Matei Zaharia. https://arxiv.org/abs/2004.12832

📄 ColBERTv2: "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction" by Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. https://arxiv.org/abs/2112.01488

📄 PLAID: "An Efficient Engine for Late Interaction Retrieval" by Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. https://arxiv.org/abs/2205.09707

📄 CEDR: "CEDR: Contextualized Embeddings for Document Ranking" by Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. https://arxiv.org/abs/1904.07094

🪃 Feedback form: https://scastella.typeform.com/to/rg7a5GfJ

Timestamps:

00:00 Introduction

00:42 Why ColBERT?

03:34 Retrieval paradigms recap

08:04 ColBERT query formulation and architecture

09:04 Using ColBERT as a reranker or as an end-to-end retriever

11:28 Space Footprint vs. MRR on MS MARCO

12:24 Methodology: datasets and negative sampling

14:37 Terminology for cross encoders, interaction-based models, etc.

16:12 Results (ColBERT v1) on MS MARCO

18:41 Ablations on model components

20:34 Max pooling vs. mean pooling

22:54 Why did ColBERT have a big impact?

26:31 ColBERTv2: knowledge distillation

29:34 ColBERTv2: indexing improvements

33:59 Effects of clustering compression in performance

35:19 Results (ColBERT v2): MS MARCO

38:54 Results (ColBERT v2): BEIR

41:27 Takeaway: strong specially in out-of-domain evaluation

43:59 Qualitatively how do ColBERT scores look like?

46:21 What's the most promising of all current neural IR paradigms

49:34 How come there's still so much interest in Dense retrieval?

51:09 Many to many similarity at different granularities

53:44 What would ColBERT v3 include?

56:39 PLAID: An Efficient Engine for Late Interaction Retrieval

Contact: castella@zeta-alpha.com