Data Engineering Podcast

How Generative AI Is Impacting Data Engineering Teams

Jul 21, 2024

Lior Gavish, co-founder of Monte Carlo, discusses how data teams are evolving to support AI features and incorporating AI into their work. Topics include impact of generative AI, integrating AI models into data workflows, challenges in building with generative AI, and optimizing AI solutions for data teams.

54:45

Creator website

Episode guests

Lior Gavish

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Generative AI enables extracting structured data from unstructured sources for deeper insights.

Data teams are incorporating AI into pipeline design to democratize data access and unlock revenue opportunities.

Deep dives

Impact of AI on Data Engineers

AI, particularly generative models, is transforming the role of data engineers. It allows for the extraction of structured data from unstructured sources like text, enabling deeper insights. Data engineering teams are finding innovative ways to apply AI, such as scouting players in professional sports using sentiment analysis on scout reports or automating revenue generation for energy companies by processing energy supply requests. This shift brings data engineers to the forefront of decision-making processes in organizations.

Intro

2min

Impact of Generative AI on Data Engineering Teams

23min

Integrating AI and Machine Learning Models into Data Engineering Workflows

9min

Impact of AI on Data Engineering Teams

11min

Challenges and Lessons Learned in Building with Generative AI

3min

Optimizing AI Solutions for Data Engineering Teams

5min

Discussing the Integration of Data Engineering and Software Engineering Stacks

2min

Summary
Generative AI has rapidly gained adoption for numerous use cases. To support those applications, organizational data platforms need to add new features and data teams have increased responsibility. In this episode Lior Gavish, co-founder of Monte Carlo, discusses the various ways that data teams are evolving to support AI powered features and how they are incorporating AI into their work.
Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Trusted by teams of all sizes, including Comcast and Doordash. Want to see Starburst in action? Go to dataengineeringpodcast.com/starburst and get $500 in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino.
Your host is Tobias Macey and today I'm interviewing Lior Gavish about the impact of AI on data engineers

Interview

Introduction
How did you get involved in the area of data management?
Can you start by clarifying what we are discussing when we say "AI"?
Previous generations of machine learning (e.g. deep learning, reinforcement learning, etc.) required new features in the data platform. What new demands is the current generation of AI introducing?
Generative AI also has the potential to be incorporated in the creation/execution of data pipelines. What are the risk/reward tradeoffs that you have seen in practice?
- What are the areas where LLMs have proven useful/effective in data engineering?
Vector embeddings have rapidly become a ubiquitous data format as a result of the growth in retrieval augmented generation (RAG) for AI applications. What are the end-to-end operational requirements to support this use case effectively?
- As with all data, the reliability and quality of the vectors will impact the viability of the AI application. What are the different failure modes/quality metrics/error conditions that they are subject to?
As much as vectors, vector databases, RAG, etc. seem exotic and new, it is all ultimately shades of the same work that we have been doing for years. What are the areas of overlap in the work required for running the current generation of AI, and what are the areas where it diverges?
- What new skills do data teams need to acquire to be effective in supporting AI applications?
What are the most interesting, innovative, or unexpected ways that you have seen AI impact data engineering teams?
What are the most interesting, unexpected, or challenging lessons that you have learned while working with the current generation of AI?
When is AI the wrong choice?
What are your predictions for the future impact of AI on data engineering teams?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.

Data Engineering Podcast

How Generative AI Is Impacting Data Engineering Teams

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

Deep dives

Impact of AI on Data Engineers

Bringing Generative AI Into Data Workflows

Challenges and Lessons in Leveraging AI

Future Trends and Impact of AI in Data Engineering

Remember Everything You Learn from Podcasts