Bringing AI Into The Inner Loop of Data Engineering With Ascend
Mar 24, 2025
auto_awesome
Sean Knapp, Founder and CEO of Ascend.io, shares his expertise in data engineering and AI's transformative role. He discusses how AI can streamline workflows, alleviate burdens for data engineers, and enhance productivity by automating tasks. Sean highlights challenges like data governance and the integration of AI into existing systems. The conversation also touches on bridging the gap between junior and senior engineers using AI as a collaborative tool, as well as the future potential of AI to revolutionize data engineering processes.
AI is revolutionizing data engineering by automating repetitive tasks and enabling data engineers to focus on more complex, innovative work.
While current AI tools offer improvements, many data engineers find them insufficient in addressing core workflow challenges and inefficiencies.
Organizations must balance the integration of AI-driven solutions with legacy systems to enhance data maturity without overwhelming their teams.
Deep dives
The Challenge of Data Migrations
Data migrations are often lengthy and resource-intensive processes that can negatively affect team morale. Organizations frequently tackle migrations that take months, if not years, to complete, which can lead to frustration among team members. To address this issue, new tools powered by artificial intelligence, such as those from Datafold, claim to speed up these migrations by a factor of up to ten times compared to traditional methods. These advancements not only promise a more efficient migration process but also include guarantees on timelines to enhance confidence among teams.
The Role of AI in Data Engineering
AI is increasingly spotlighted as a game-changer in data engineering, with companies like Ascend implementing advanced automation on their platforms. By focusing on not just repetitive tasks, but also complex data pipeline creation, Ascend aims to alleviate the burdens faced by data engineering teams. Despite a general excitement surrounding AI, many data engineers still find their tools lacking as they grapple with inefficient systems that hinder productivity. This gap presents an opportunity for innovation and improvements in the dynamics of data engineering work processes.
The Limitations of Current AI Integration
While there are innovative uses of AI in software engineering, data engineering has yet to experience its own transformative moment akin to that of ChatGPT. Many data teams view current AI tools as minor enhancements rather than groundbreaking innovations that can drastically change workflows. This hesitance may come from the perception that existing AI integrations do not address the core challenges of data engineering itself, such as writing code, testing, and maintaining operational visibility. Recognizing this, industry leaders are exploring ways to develop more intuitive, powerful AI tools tailored to the unique demands of data engineers.
Navigating the Shift to AI-Powered Data Solutions
As organizations strive to incorporate AI into their operations, they face pressure to accelerate their data maturity cycles without the necessary foundational tools. This often results in overwhelmed teams trying to evolve from basic data management systems to robust AI-driven solutions too quickly. To manage this, many are now seeking to utilize AI not just for process automation but also for managing legacy systems and workflows. AI's potential to streamline these transitions can provide significant leverage for data teams and facilitate smoother integrations into modern architectures.
Future Directions and Innovations in Data Engineering
With ongoing advancements in AI and an increasing emphasis on automation, organizations can expect to see a significant shift in data engineering practices. Future developments may focus on creating intelligent systems that provide enhanced context and insights, reducing manual intervention in routine operations. Companies are beginning to emphasize the creation of customized agents that work alongside data teams to address specific needs and automate much of the toil involved in data management. As the technology landscape evolves, those who embrace these changes may find themselves at a major advantage in adapting to the fast-paced demands of the industry.
Summary In this episode of the Data Engineering Podcast Sean Knapp, CEO of Ascend.io, explores the intersection of AI and data engineering. He discusses the evolution of data engineering and the role of AI in automating processes, alleviating burdens on data engineers, and enabling them to focus on complex tasks and innovation. The conversation covers the challenges and opportunities presented by AI, including the need for intelligent tooling and its potential to streamline data engineering processes. Sean and Tobias also delve into the impact of generative AI on data engineering, highlighting its ability to accelerate development, improve governance, and enhance productivity, while also noting the current limitations and future potential of AI in the field.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm interviewing Sean Knapp about how Ascend is incorporating AI into their platform to help you keep up with the rapid rate of change
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what Ascend is and the story behind it?
The last time we spoke was August of 2022. What are the most notable or interesting evolutions in your platform since then?
In that same time "AI" has taken up all of the oxygen in the data ecosystem. How has that impacted the ways that you and your customers think about their priorities?
The introduction of AI as an API has caused many organizations to try and leap-frog their data maturity journey and jump straight to building with advanced capabilities. How is that impacting the pressures and priorities felt by data teams?
At the same time that AI-focused product goals are straining data teams capacities, AI also has the potential to act as an accelerator to their work. What are the roadblocks/speedbumps that are in the way of that capability?
Many data teams are incorporating AI tools into parts of their workflow, but it can be clunky and cumbersome. How are you thinking about the fundamental changes in how your platform works with AI at its center?
Can you describe the technical architecture that you have evolved toward that allows for AI to drive the experience rather than being a bolt-on?
What are the concrete impacts that these new capabilities have on teams who are using Ascend?
What are the most interesting, innovative, or unexpected ways that you have seen Ascend + AI used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on incorporating AI into the core of Ascend?
When is Ascend the wrong choice?
What do you have planned for the future of AI in Ascend?
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
