How AI Is Built

Mór Kapronczay, Head of ML at Superlinked, unpacks the nuances of embeddings beyond just text. He emphasizes that traditional text embeddings fall short, especially with complex data. Mór introduces multi-modal embeddings that integrate various data types, improving search relevance and user experiences. He also discusses challenges in embedding numerical data, suggesting innovative methods like logarithmic transformations. The conversation delves into balancing speed and accuracy in vector searches, highlighting the dynamic nature of real-time data prioritization.

Today we have Jessica Talisman with us, who is working as an Information Architect at Adobe. She is (in my opinion) the expert on taxonomies and ontologies.That’s what you will learn today in this episode of How AI Is Built. Taxonomies, ontologies, knowledge graphs.Everyone is talking about them no-one knows how to build them.But before we look into that, what are they good for in search?Imagine a large corpus of academic papers. When a user searches for "machine learning in healthcare", the system can:Recognize "machine learning" as a subcategory of "artificial intelligence"Identify "healthcare" as a broad field with subfields like "diagnostics" and "patient care"We can use these to expand the query or narrow it down.We can return results that include papers on "neural networks for medical imaging" or "predictive analytics in patient outcomes", even if these exact phrases weren't in the search queryWe can also filter down and remove papers not tagged with AI that might just mention it in a side not.So we are building the plumbing, the necessary infrastructure for tagging, categorization, query expansion and relexation, filtering.So how can we build them?1️⃣ Start with Industry Standards • Leverage established taxonomies (e.g., Google, GS1, IAB) • Audit them for relevance to your project • Use as a foundation, not a final solution2️⃣ Customize and Fill Gaps • Adapt industry taxonomies to your specific domain • Create a "coverage model" for your unique needs • Mine internal docs to identify domain-specific concepts3️⃣ Follow Ontology Best Practices • Use clear, unique primary labels for each concept • Include definitions to avoid ambiguity • Provide context for each taxonomy nodeJessica Talisman:LinkedInNicolay Gerold:⁠LinkedIn⁠⁠X (Twitter)00:00 Introduction to Taxonomies and Knowledge Graphs 02:03 Building the Foundation: Metadata to Knowledge Graphs 04:35 Industry Taxonomies and Coverage Models 06:32 Clustering and Labeling Techniques 11:00 Evaluating and Maintaining Taxonomies 31:41 Exploring Taxonomy Granularity 32:18 Differentiating Taxonomies for Experts and Users 33:35 Mapping and Equivalency in Taxonomies 34:02 Best Practices and Examples of Taxonomies 40:50 Building Multilingual Taxonomies 44:33 Creative Applications of Taxonomies 48:54 Overrated and Underappreciated Technologies 53:00 The Importance of Human Involvement in AI 53:57 Connecting with the Speaker 55:05 Final Thoughts and Takeaways

Jo Bergum, Chief Scientist at Vespa, dives into the game-changing technology of ColPali, which revolutionizes document processing by merging late interaction scoring and visual language models. He discusses how ColPali effectively handles messy data, allowing for seamless searches across complex formats like PDFs and HTML. By eliminating the need for extensive text extraction, ColPali enhances both efficiency and user experience. Its applications span multiple domains, promising significant advancements in information retrieval technology.

Aamir Shakir, founder of mixedbread.ai, is an expert in crafting advanced embedding and reranking models for search applications. He discusses the transformative power of rerankers in retrieval systems, emphasizing their role in enhancing search relevance and performance without complete overhauls. Aamir highlights the benefits of late interaction models like ColBERT for better interpretability and shares creative applications of rerankers beyond traditional use. He also navigates future challenges in multimodal data management and the exciting possibilities of compound models for unified search.

Join Nils Reimers, a prominent researcher in dense embeddings and the driving force behind foundational search models at Cohere. He dives into the intriguing limitations of text embeddings, such as their struggles with long documents and out-of-domain data. Reimers shares insights on the necessity of fine-tuning to adapt models effectively. He also discusses innovative approaches like re-ranking to enhance search relevance, and the bright future of embeddings as new research avenues are explored. Don't miss this deep dive into the cutting-edge of AI!

Nirant Kasliwal, an author known for his expertise in metadata extraction and evaluation strategies, shares invaluable insights on scaling Retrieval-Augmented Generation (RAG) systems. He dives into common pitfalls such as the challenges posed by naive RAG and the sensitivity of LLMs to input. Strategies for query profiling, user personalization, and effective metadata extraction are discussed. Nirant emphasizes the importance of understanding user context to deliver precise information, ultimately aiming to enhance the efficiency of RAG implementations.

Doug Turnbull, a search engineer at Reddit and author of "Relevant Search," dives into the transformation of search from keyword basics to advanced methods like semantic search and LLMs. He highlights the ongoing challenges of defining relevance based on user intent and context. Doug also discusses the importance of integrating various search techniques for better results, emphasizing the role of operational concerns in shaping search technology. With insights on the resurgence of underappreciated methods like LambdaMART, he shares how understanding user perspectives can significantly enhance search performance.

Charlie Hull, a search expert and the founder of Flax, dives into the world of data-driven search optimization. He discusses the challenges of measuring relevance in search, emphasizing its subjective nature. Common pitfalls in search assessments are highlighted, including overvaluing speed and user complaints. Hull shares effective methods for evaluating search systems, such as human evaluation and user interaction analysis. He also explores the balancing act between business goals and user needs, and the crucial role of data quality in delivering optimal search results.

Join Daniel Tunkelang, a seasoned search consultant and leader in AI-powered search, as he explores the nuances of query understanding. He emphasizes that the user's query is paramount and advocates for a proactive approach to enhancing search systems. Discover the significance of query specificity, the advantages of classifying queries, and how simpler techniques can rival complex models. Tunkelang also shares insights on optimizing query processing and the challenges of categorizing data in an ever-evolving landscape.

Today we are launching the season 2 of How AI Is Built.The last few weeks, we spoke to a lot of regular listeners and past guests and collected feedback. Analyzed our episode data. And we will be applying the learnings to season 2.This season will be all about search.We are trying to make it better, more actionable, and more in-depth. The goal is that at the end of this season, you have a full-fleshed course on search in podcast form, which mini-courses on specific elements like RAG.We will be talking to experts from information retrieval, information architecture, recommendation systems, and RAG; from academia and industry. Fields that do not really talk to each other.We will try to unify and transfer the knowledge and give you a full tour of search, so you can build your next search application or feature with confidence.We will be talking to Charlie Hull on how to systematically improve search systems, with Nils Reimers on the fundamental flaws of embeddings and how to fix them, with Daniel Tunkelang on how to actually understand the queries of the user, and many more.We will try to bridge the gaps. How to use decades of research and practice in iteratively improving traditional search and apply it to RAG. How to take new methods from recommendation systems and vector databases and bring it into traditional search systems. How to use all of the different methods as search signals and combine them to deliver the results your user actually wants.We will be using two types of episodes:Traditional deep dives, like we have done them so far. Each one will dive into one specific topic within search interviewing an expert on that topic.Supplementary episodes, which answer one additional question; often either complementary or precursory knowledge for the episode, which we did not get to in the deep dive.We will be starting with episodes next week, looking at the first, last, and overarching action in search: understanding user intent and understanding the queries with Daniel Tunkelang.I am really excited to kick this off.I would love to hear from you:What would you love to learn in this season?What guest should I have on?What topics should I make a deep dive on (try to be specific)?Yeah, let me know in the comments or just slide into my DMs on Twitter or LinkedIn.I am looking forward to hearing from you guys.I want to try to be more interactive. So anytime you encounter anything unclear or any question pops up in one of the episode, give me a shout and I will try to answer it to you and to everyone.Enough of me rambling. Let’s kick this off. I will see you next Thursday, when we start with query understanding.Shoot me a message and stay up to date:⁠LinkedIn⁠⁠X (Twitter)