Supporting And Expanding The Arrow Ecosystem For Fast And Efficient Data Processing At Voltron Data
Data Engineering Podcast
00:00
The Biggest Gap in Data Management
I think we're still in a state of learning and change when it comes to how best to build and manage very large data lakes. I'm really hopeful about Apache iceberg, which came out of Netflix and is one of the next generation approaches for large scale data set management. The sooner that we can settle on standards for scalable open data warehousing, so to speak, I think that makes things less of a moving target. And so the world becomes increasingly standardized on iceberg, for example, in file formats like parquet,. Then that simplifies the problem for the engine and user interface developers who want to make an end-to-end stack.
Transcript
Play full episode
Transcript
Episode notes
Summary
The data ecosystem has been growing rapidly, with new communities joining and bringing their preferred programming languages to the mix. This has led to inefficiencies in how data is stored, accessed, and shared across process and system boundaries. The Arrow project is designed to eliminate wasted effort in translating between languages, and Voltron Data was created to help grow and support its technology and community. In this episode Wes McKinney shares the ways that Arrow and its related projects are improving the efficiency of data systems and driving their next stage of evolution.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
- Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos.
- Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the leading end-to-end Data Observability Platform! Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, dbt models, Airflow jobs, and business intelligence tools, reducing time to detection and resolution from weeks to just minutes. Monte Carlo also gives you a holistic picture of data health with automatic, end-to-end lineage from ingestion to the BI layer directly out of the box. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/montecarlo to learn more.
- Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc., are already available as plug-and-play connectors with reliable, intuitive SaaS solutions. Hevo Data is a highly reliable and intuitive data pipeline platform used by data engineers from 40+ countries to set up and run low-latency ELT pipelines with zero maintenance. Boasting more than 150 out-of-the-box connectors that can be set up in minutes, Hevo also allows you to monitor and control your pipelines. You get: real-time data flow visibility, fail-safe mechanisms, and alerts if anything breaks; preload transformations and auto-schema mapping precisely control how data lands in your destination; models and workflows to transform data for analytics; and reverse-ETL capability to move the transformed data back to your business software to inspire timely action. All of this, plus its transparent pricing and 24*7 live support, makes it consistently voted by users as the Leader in the Data Pipeline category on review platforms like G2. Go to dataengineeringpodcast.com/hevodata and sign up for a free 14-day trial that also comes with 24×7 support.
- Your host is Tobias Macey and today I’m interviewing Wes McKinney about his work at Voltron Data and on the Arrow ecosystem
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what you are building at Voltron Data and the story behind it?
- What is the vision for the broader data ecosystem that you are trying to realize through your investment in Arrow and related projects?
- How does your work at Voltron Data contribute to the realization of that vision?
- What is the impact on engineer productivity and compute efficiency that gets introduced by the impedance mismatches between language and framework representations of data?
- The scope and capabilities of the Arrow project have grown substantially since it was first introduced. Can you give an overview of the current features and extensions to the project?
- What are some of the ways that ArrowVe and its related projects can be integrated with or replace the different elements of a data platform?
- Can you describe how Arrow is implemented?
- What are the most complex/challenging aspects of the engineering needed to support interoperable data interchange between language runtimes?
- How are you balancing the desire to move quickly and improve the Arrow protocol and implementations, with the need to wait for other players in the ecosystem (e.g. database engines, compute frameworks, etc.) to add support?
- With the growing application of data formats such as graphs and vectors, what do you see as the role of Arrow and its ideas in those use cases?
- For workflows that rely on integrating structured and unstructured data, what are the options for interaction with non-tabular data? (e.g. images, documents, etc.)
- With your support-focused business model, how are you approaching marketing and customer education to make it viable and scalable?
- What are the most interesting, innovative, or unexpected ways that you have seen Arrow used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Arrow and its ecosystem?
- When is Arrow the wrong choice?
- What do you have planned for the future of Arrow?
Contact Info
- Website
- wesm on GitHub
- @wesmckinn on Twitter
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
- Voltron Data
- Pandas
- Apache Arrow
- Partial Differential Equation
- FPGA == Field-Programmable Gate Array
- GPU == Graphics Processing Unit
- Ursa Labs
- Voltron (cartoon)
- Feature Engineering
- PySpark
- Substrait
- Arrow Flight
- Acero
- Arrow Datafusion
- Velox
- Ibis
- SIMD == Single Instruction, Multiple Data
- Lance
- DuckDB
- Data Threads Conference
- Nano-Arrow
- Arrow ADBC Protocol
- Apache Iceberg
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
- Atlan:  Have you ever woken up to a crisis because a number on a dashboard is broken and no one knows why? Or sent out frustrating slack messages trying to find the right data set? Or tried to understand what a column name means? Our friends at Atlan started out as a data team themselves and faced all this collaboration chaos themselves, and started building Atlan as an internal tool for themselves. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to [dataengineeringpodcast.com/atlan](https://www.dataengineeringpodcast.com/atlan) and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription.
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
🤯 How amazing is this app?!
Sophie
App Store
Snipd helps me keep track of everything I resonated with from a show. I can always find my snips, listen to them again, share them. And there are so many great product updates regularly! Like the feature to chat with the episode: I can retrieve even more knowledge from an episode 🥹 I’m in love with this app 🤍
Best podcast app
JD Stuart
App Store
I’ve been using Snipd for about a year and this app has been life changing. I listen to about 17 hours of podcasts a week and I want to take notes on 95% of them. Snipd makes it so easy to do. I can triple click my headphones and record a snip. The app also improves rapidly which is welcomed. It’s an easy subscription for me to pay.