Micah Wylde from Arroyo discusses their real-time data processing engine using Rust, enabling SQL queries on streaming data. They explore Rust's benefits in performance, memory safety, Tokyo framework usage, and challenges in Rust engineering. The episode delves into the future of Rust ecosystem, interaction with other languages, limitations, and industry perceptions.
Arroyo simplifies stream processing by enabling SQL queries with Rust user-defined functions for real-time data insights.
Rust usage in data systems, like Arroyo, showcases the trend of adopting Rust for enhanced performance in stream processing applications.
Deep dives
The Significance of Using Rust in Data Systems
The usage of Rust in data systems, especially for stream processing, has gained significant traction due to its efficiency and performance advantages over languages like Java. Companies like Arroyo and InfluxDB have successfully rewritten parts of their systems in Rust to enhance performance and scalability, showcasing the growing trend of adopting Rust in this space.
Streamlining Data Processing with Rust in Arroyo
Arroyo, a real-time data processing engine, simplifies stream processing for data engineers by enabling SQL queries with Rust user-defined functions. This innovative approach allows for seamless data processing in real-time scenarios, providing faster insights and analysis compared to traditional batch processing methods.
The Evolution of Stream Processing Technologies
The evolution of stream processing technologies like Apache Flink showcases the shift towards processing data in real-time to enable instant insights and analytics. Innovations like Arroyo aim to address the limitations of existing systems by offering a more user-friendly and efficient solution that empowers engineers of various backgrounds to build real-time data pipelines.
Future Trends and Challenges in the Rust Ecosystem
Looking ahead, the Rust ecosystem is poised for further growth, particularly in the data science and application software domains. As Rust matures and continues to optimize its features, the industry may witness increased adoption of Rust in various software development projects, propelling the language into new frontiers of innovation and efficiency.
About Arroyo Arroyo was founded in 2022 by Micah Wylde and is based in San Francisco, CA. It is backed by Y Combinator (https://www.ycombinator.com/) (YC W23). The companies' mission is to accelerate the transition from batch-processing to a streaming-first world.
About Micah Wylde Micah was previously tech lead for streaming compute at Splunk and Lyft, where he built real-time data infra powering Lyft's dynamic pricing, ETA, and safety features. He spends his time rock climbing, playing music, and bringing real-time data to companies that can't hire a streaming infra team.
Tools and Services Mentioned - Apache Flink: https://flink.apache.org/ - Tokio Discord: https://discord.gg/tokio - Clippy: https://github.com/rust-lang/rust-clippy - Zero to Production in Rust by Luca Palmieri: https://www.zero2prod.com/ - Apache DataFusion: https://github.com/apache/arrow-datafusion - Axum web framework: https://github.com/tokio-rs/axum - `sqlx` crate: https://github.com/launchbadge/sqlx - `log` crate: https://github.com/rust-lang/log - `tokio tracing` crate: https://github.com/tokio-rs/tracing - wasmtime - A standalone runtime for WebAssembly: https://github.com/bytecodealliance/wasmtime
References To Other Episodes - Rust in Production Season 1 Episode 1: InfluxData: https://corrode.dev/podcast/s01e01-influxdata
Official Links - Arroyo Homepage: https://www.arroyo.dev/ - Arroyo Streaming Engine: https://github.com/ArroyoSystems/arroyo - Blog Post: Rust Is The Best Language For Data Infra: https://www.arroyo.dev/blog/rust-for-data-infra - Micah Wylde on LinkedIn: https://www.linkedin.com/in/wylde/ - Micah Wylde on GitHub: https://github.com/mwylde - Micah Wylde's Personal Homepage: https://www.micahw.com/
Get the Snipd podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share & Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered podcast player
Listen to all your favourite podcasts with AI-powered features
Discover highlights
Listen to the best highlights from the podcasts you love and dive into the full episode