Streamlining Data Pipelines with MCP Servers and Vector Engines

10 snips

Jul 15, 2025

Kacper Łukawski, a Senior Developer Advocate at Qdrant, specializes in vector databases for large language models. He dives into transforming unstructured data into valuable insights through semantic search and retrieval-augmented generation. Kacper explains the integration of MCP servers for optimizing data pipelines and discusses the challenges of managing embeddings. He also highlights innovative applications in coding practices and the complexities of vector search, offering practical advice on fine-tuning models and reducing costs for enhanced search quality.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

Kacper's Big Data Beginnings

Kacper Łukawski started working with big data pipelines around 2014-15 in the automotive industry.
He used Apache tools like Spark, Kafka, and Uzi, expanding into data ingestion and visualization projects.

INSIGHT

LLMs Are Not Magic Fixes

Large language models (LLMs) do not automatically fix poor data quality or solve all data problems.
Teams often struggle with scalability and deployment, especially in industries avoiding proprietary SaaS tools.

ANECDOTE

Ontology via Vector and Graph

One user built a system combining vector embeddings and graphs to derive ontologies in law and medicine.
This dual modeling approach captures semantics and relationships, popular in graph-RAG scenarios.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Summary
In this episode of the Data Engineering Podcast Kacper Łukawski from Qdrant about integrating MCP servers with vector databases to process unstructured data. Kacper shares his experience in data engineering, from building big data pipelines in the automotive industry to leveraging large language models (LLMs) for transforming unstructured datasets into valuable assets. He discusses the challenges of building data pipelines for unstructured data and how vector databases facilitate semantic search and retrieval-augmented generation (RAG) applications. Kacper delves into the intricacies of vector storage and search, including metadata and contextual elements, and explores the evolution of vector engines beyond RAG to applications like semantic search and anomaly detection. The conversation covers the role of Model Context Protocol (MCP) servers in simplifying data integration and retrieval processes, highlighting the need for experimentation and evaluation when adopting LLMs, and offering practical advice on optimizing vector search costs and fine-tuning embedding models for improved search quality.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm interviewing Kacper Łukawski about how MCP servers can be paired with vector databases to streamline processing of unstructured data

Interview

Introduction
How did you get involved in the area of data management?
LLMs are enabling the derivation of useful data assets from unstructured sources. What are the challenges that teams face in building the pipelines to support that work?
How has the role of vector engines grown or evolved in the past ~2 years as LLMs have gained broader adoption?
- Beyond its role as a store of context for agents, RAG, etc. what other applications are common for vector databaes?
In the ecosystem of vector engines, what are the distinctive elements of Qdrant?
How has the MCP specification simplified the work of processing unstructured data?
Can you describe the toolchain and workflow involved in building a data pipeline that leverages an MCP for generating embeddings?
helping data engineers gain confidence in non-deterministic workflows
bringing application/ML/data teams into collaboration for determining the impact of e.g. chunking strategies, embedding model selection, etc.
What are the most interesting, innovative, or unexpected ways that you have seen MCP and Qdrant used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on vector use cases?
When is MCP and/or Qdrant the wrong choice?
What do you have planned for the future of MCP with Qdrant?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA