Andrew Luo, CEO of OneSchema, shares his expertise in data engineering and CRM migration, focusing on the enduring relevance of CSVs. He discusses the common challenges of CSV data, such as inconsistency and lack of standards, and explains how OneSchema uses AI for improved type validation and parsing. Andrew highlights OneSchema's potential to streamline data imports and boost efficiency, particularly for non-technical users. He also reveals plans for future innovations, including industry-specific transformation packs to enhance data management further.
54:40
forum Ask episode
web_stories AI Snips
view_agenda Chapters
auto_awesome Transcript
info_circle Episode notes
insights INSIGHT
CSV Ubiquity
CSVs are ubiquitous because they are the 'lingua franca' of data exchange, easily understood by most business users.
They simplify cross-company data sharing, unlike APIs that require custom integration.
insights INSIGHT
CSV Security
CSVs are often preferred for sensitive data like government, healthcare, legal, and financial data due to perceived security and control.
Senders feel more secure knowing exactly what's being sent, unlike constantly changing APIs.
question_answer ANECDOTE
Cross-Company Data Exchange
Cross-company data exchanges often rely on spreadsheets due to the involvement of business users in schema mapping.
These users need easily understandable formats for data discovery and prep, making spreadsheets a natural choice.
Get the Snipd Podcast app to discover more snips from this episode
Summary In this episode of the Data Engineering Podcast Andrew Luo, CEO of OneSchema, talks about handling CSV data in business operations. Andrew shares his background in data engineering and CRM migration, which led to the creation of OneSchema, a platform designed to automate CSV imports and improve data validation processes. He discusses the challenges of working with CSVs, including inconsistent type representation, lack of schema information, and technical complexities, and explains how OneSchema addresses these issues using multiple CSV parsers and AI for data type inference and validation. Andrew highlights the business case for OneSchema, emphasizing efficiency gains for companies dealing with large volumes of CSV data, and shares plans to expand support for other data formats and integrate AI-driven transformation packs for specific industries.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm interviewing Andrew Luo about how OneSchema addresses the headaches of dealing with CSV data for your business
Interview
Introduction
How did you get involved in the area of data management?
Despite the years of evolution and improvement in data storage and interchange formats, CSVs are just as prevalent as ever. What are your opinions/theories on why they are so ubiquitous?
What are some of the major sources of CSV data for teams that rely on them for business and analytical processes?
The most obvious challenge with CSVs is their lack of type information, but they are notorious for having numerous other problems. What are some of the other major challenges involved with using CSVs for data interchange/ingestion?
Can you describe what you are building at OneSchema and the story behind it?
What are the core problems that you are solving, and for whom?
Can you describe how you have architected your platform to be able to manage the variety, volume, and multi-tenancy of data that you process?
How have the design and goals of the product changed since you first started working on it?
What are some of the major performance issues that you have encountered while dealing with CSV data at scale?
What are some of the most surprising things that you have learned about CSVs in the process of building OneSchema?
What are the most interesting, innovative, or unexpected ways that you have seen OneSchema used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on OneSchema?
When is OneSchema the wrong choice?
What do you have planned for the future of OneSchema?
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
Data Engineering Podcast