CSVs Will Never Die And OneSchema Is Counting On It
Data Engineering Podcast
00:00
Navigating the Limitations of CSVs in Data Management
This chapter explores the complexities of using CSV files for data representation, particularly in type inference and schema definition. It also highlights a company's innovative approach to enhancing CSV functionality through data type validation and AI-driven insights.
Transcript
Play full episode
Transcript
Episode notes
Summary
In this episode of the Data Engineering Podcast Andrew Luo, CEO of OneSchema, talks about handling CSV data in business operations. Andrew shares his background in data engineering and CRM migration, which led to the creation of OneSchema, a platform designed to automate CSV imports and improve data validation processes. He discusses the challenges of working with CSVs, including inconsistent type representation, lack of schema information, and technical complexities, and explains how OneSchema addresses these issues using multiple CSV parsers and AI for data type inference and validation. Andrew highlights the business case for OneSchema, emphasizing efficiency gains for companies dealing with large volumes of CSV data, and shares plans to expand support for other data formats and integrate AI-driven transformation packs for specific industries.
Announcements
Parting Question
In this episode of the Data Engineering Podcast Andrew Luo, CEO of OneSchema, talks about handling CSV data in business operations. Andrew shares his background in data engineering and CRM migration, which led to the creation of OneSchema, a platform designed to automate CSV imports and improve data validation processes. He discusses the challenges of working with CSVs, including inconsistent type representation, lack of schema information, and technical complexities, and explains how OneSchema addresses these issues using multiple CSV parsers and AI for data type inference and validation. Andrew highlights the business case for OneSchema, emphasizing efficiency gains for companies dealing with large volumes of CSV data, and shares plans to expand support for other data formats and integrate AI-driven transformation packs for specific industries.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
- Your host is Tobias Macey and today I'm interviewing Andrew Luo about how OneSchema addresses the headaches of dealing with CSV data for your business
- Introduction
- How did you get involved in the area of data management?
- Despite the years of evolution and improvement in data storage and interchange formats, CSVs are just as prevalent as ever. What are your opinions/theories on why they are so ubiquitous?
- What are some of the major sources of CSV data for teams that rely on them for business and analytical processes?
- The most obvious challenge with CSVs is their lack of type information, but they are notorious for having numerous other problems. What are some of the other major challenges involved with using CSVs for data interchange/ingestion?
- Can you describe what you are building at OneSchema and the story behind it?
- What are the core problems that you are solving, and for whom?
- Can you describe how you have architected your platform to be able to manage the variety, volume, and multi-tenancy of data that you process?
- How have the design and goals of the product changed since you first started working on it?
- What are some of the major performance issues that you have encountered while dealing with CSV data at scale?
- What are some of the most surprising things that you have learned about CSVs in the process of building OneSchema?
- What are the most interesting, innovative, or unexpected ways that you have seen OneSchema used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on OneSchema?
- When is OneSchema the wrong choice?
- What do you have planned for the future of OneSchema?
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
- OneSchema
- EDI == Electronic Data Interchange
- UTF-8 BOM (Byte Order Mark) Characters
- SOAP
- CSV RFC
- Iceberg
- SSIS == SQL Server Integration Services
- MS Access
- Datafusion
- JSON Schema
- SFTP == Secure File Transfer Protocol
The AI-powered Podcast Player
Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
