Accelerate Migration Of Your Data Warehouse with Datafold's AI Powered Migration Agent

19 snips

Oct 27, 2024

Gleb Mezhanskiy, CEO and co-founder of DataFold, shares his extensive experience in data management from his time at Autodesk and Lyft. He dives into the complexities of data migrations, detailing challenges like technical debt and the need for effective parity between systems. Gleb reveals how DataFold leverages AI to automate data migration processes, significantly reducing time and effort. He also discusses the importance of monitoring data integrity in real-time and offers insights into choosing the right models for secure data handling.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

Lyft's Migration Struggle

Gleb Mezhanskiy's data migration at Lyft, from Redshift to a data lake, took five years instead of nine months.
This painful experience, including process and tooling mistakes, inspired him to improve migrations.

ADVICE

Lift and Shift

Prioritize lift-and-shift migrations, minimizing changes to data models or definitions.
Focus on replicating the legacy system on the new platform to achieve parity quickly.

INSIGHT

Parity is Key

Data migration success hinges on business trust, achieved by demonstrating parity between old and new systems.
The bulk of migration time (80%) is spent on translating business logic and reconciling outputs.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Summary
Gleb Mezhanskiy, CEO and co-founder of DataFold, joins Tobias Macey to discuss the challenges and innovations in data migrations. Gleb shares his experiences building and scaling data platforms at companies like Autodesk and Lyft, and how these experiences inspired the creation of DataFold to address data quality issues across teams. He outlines the complexities of data migrations, including common pitfalls such as technical debt and the importance of achieving parity between old and new systems. Gleb also discusses DataFold's innovative use of AI and large language models (LLMs) to automate translation and reconciliation processes in data migrations, reducing time and effort required for migrations.
Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Imagine catching data issues before they snowball into bigger problems. That’s what Datafold’s new Monitors do. With automatic monitoring for cross-database data diffs, schema changes, key metrics, and custom data tests, you can catch discrepancies and anomalies in real time, right at the source. Whether it’s maintaining data integrity or preventing costly mistakes, Datafold Monitors give you the visibility and control you need to keep your entire data stack running smoothly. Want to stop issues before they hit production? Learn more at dataengineeringpodcast.com/datafold today!
Your host is Tobias Macey and today I'm welcoming back Gleb Mezhanskiy to talk about Datafold's experience bringing AI to bear on the problem of migrating your data stack

Interview

Introduction
How did you get involved in the area of data management?
Can you describe what the Data Migration Agent is and the story behind it?
- What is the core problem that you are targeting with the agent?
What are the biggest time sinks in the process of database and tooling migration that teams run into?
Can you describe the architecture of your agent?
- What was your selection and evaluation process for the LLM that you are using?
What were some of the main unknowns that you had to discover going into the project?
- What are some of the evolutions in the ecosystem that occurred either during the development process or since your initial launch that have caused you to second-guess elements of the design?
In terms of SQL translation there are libraries such as SQLGlot and the work being done with SDF that aim to address that through AST parsing and subsequent dialect generation. What are the ways that approach is insufficient in the context of a platform migration?
How does the approach you are taking with the combination of data-diffing and automated translation help build confidence in the migration target?
What are the most interesting, innovative, or unexpected ways that you have seen the Data Migration Agent used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on building an AI powered migration assistant?
When is the data migration agent the wrong choice?
What do you have planned for the future of applications of AI at Datafold?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA