Adding ML layer to Search: Hybrid Search Optimizer with Daniel Wrigley and Eric Pugh

Vector Podcast website: https://vectorpodcast.com

Haystack US 2025: https://haystackconf.com/2025/

Federated search, Keyword & Neural Search, ML Optimisation, Pros and Cons of Hybrid search

It is fascinating and funny how things develop, but also turn around. In 2022-23 everyone was buzzing about hybrid search. In 2024 the conversation shifted to RAG, RAG, RAG. And now we are in 2025 and back to hybrid search - on a different level: finally there are strides and contributions towards making hybrid search parameters learnt with ML. How cool is that?

Design: Saurabh Rai, https://www.linkedin.com/in/srbhr/

The design of this episode is inspired by a scene in Blade Runner 2049. There's a clear path leading towards where people want to go to, yet they're searching for something.

00:00 Intro

00:54 Eric's intro and Daniel's background

02:50 Importance of Hybrid search: Daniel's take

07:26 Eric's take

10:57 Dmitry's take

11:41 Eric's predictions

13:47 Doug's blog on RRF is not enough

16:18 How to not fall short of the blind picking in RRF: score normalization, combinations and weights

25:03 The role of query understanding: feature groups

35:11 Lesson 1 from Daniel: Simple models might be all you need

36:30 Lesson 2: query features might be all you need

38:30 Reasoning capabilities in search

40:02 Question from Eric: how is this different from Learning To Rank?

42:46 Carrying the past in Learning To Rank / any rank

44:21 Demo!

51:52 How to consume this in OpenSearch

55:15 What's next

58:44 Haystack US 2025