Vector Podcast cover image

Vector Podcast

Adding ML layer to Search: Hybrid Search Optimizer with Daniel Wrigley and Eric Pugh

Mar 21, 2025
01:03:09

Vector Podcast website: https://vectorpodcast.com

Haystack US 2025: https://haystackconf.com/2025/

Federated search, Keyword & Neural Search, ML Optimisation, Pros and Cons of Hybrid search

It is fascinating and funny how things develop, but also turn around. In 2022-23 everyone was buzzing about hybrid search. In 2024 the conversation shifted to RAG, RAG, RAG. And now we are in 2025 and back to hybrid search - on a different level: finally there are strides and contributions towards making hybrid search parameters learnt with ML. How cool is that?

Design: Saurabh Rai, https://www.linkedin.com/in/srbhr/

The design of this episode is inspired by a scene in Blade Runner 2049. There's a clear path leading towards where people want to go to, yet they're searching for something.

00:00 Intro

00:54 Eric's intro and Daniel's background

02:50 Importance of Hybrid search: Daniel's take

07:26 Eric's take

10:57 Dmitry's take

11:41 Eric's predictions

13:47 Doug's blog on RRF is not enough

16:18 How to not fall short of the blind picking in RRF: score normalization, combinations and weights

25:03 The role of query understanding: feature groups

35:11 Lesson 1 from Daniel: Simple models might be all you need

36:30 Lesson 2: query features might be all you need

38:30 Reasoning capabilities in search

40:02 Question from Eric: how is this different from Learning To Rank?

42:46 Carrying the past in Learning To Rank / any rank

44:21 Demo!

51:52 How to consume this in OpenSearch

55:15 What's next

58:44 Haystack US 2025

Get the Snipd
podcast app

Unlock the knowledge in podcasts with the podcast player of the future.
App store bannerPlay store banner

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode

Save any
moment

Hear something you like? Tap your headphones to save it with AI-generated key takeaways

Share
& Export

Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more

AI-powered
podcast player

Listen to all your favourite podcasts with AI-powered features

Discover
highlights

Listen to the best highlights from the podcasts you love and dive into the full episode