Arroyo with Micah Wylde

Jan 25, 2024

Micah Wylde from Arroyo discusses their real-time data processing engine using Rust, enabling SQL queries on streaming data. They explore Rust's benefits in performance, memory safety, Tokyo framework usage, and challenges in Rust engineering. The episode delves into the future of Rust ecosystem, interaction with other languages, limitations, and industry perceptions.

Ask episode

Chapters

Transcript

Episode notes

Introduction

00:00 • 4min

Evolution of Stream Processing Technologies

03:57 • 5min

Exploring Real-Time Processing with Apache Arrow SQL

08:34 • 3min

Exploring the Evolution of Programming Languages in Data Infrastructure Projects

11:26 • 3min

Optimization, Memory Management, and Data Flow in the Storage Layer

13:58 • 3min

Exploring Tokyo and Async Rust in Arroyo

16:34 • 14min

Evolution and Challenges in Rust Engineering

30:06 • 23min

Reflections on REST Community Drama and Platform Introduction

53:26 • 2min

About Arroyo
Arroyo was founded in 2022 by Micah Wylde and is based in San Francisco, CA. It is backed by Y Combinator (https://www.ycombinator.com/) (YC W23). The companies' mission is to accelerate the transition from batch-processing to a streaming-first world.

About Micah Wylde
Micah was previously tech lead for streaming compute at Splunk and Lyft, where he built real-time data infra powering Lyft's dynamic pricing, ETA, and safety features. He spends his time rock climbing, playing music, and bringing real-time data to companies that can't hire a streaming infra team.

Proudly Supported by CodeCrafters

CodeCrafters helps you become proficient in Rust by building real-world, production-grade projects. Learn hands-on by creating your own shell, HTTP server, Redis, Kafka, Git, SQLite, or DNS service from scratch.

Start for free today and enjoy 40% off any paid plan by using this link.

Tools and Services Mentioned
- Apache Flink: https://flink.apache.org/
- Tokio Discord: https://discord.gg/tokio
- Clippy: https://github.com/rust-lang/rust-clippy
- Zero to Production in Rust by Luca Palmieri: https://www.zero2prod.com/
- Apache DataFusion: https://github.com/apache/arrow-datafusion
- Axum web framework: https://github.com/tokio-rs/axum
- `sqlx` crate: https://github.com/launchbadge/sqlx
- `log` crate: https://github.com/rust-lang/log
- `tokio tracing` crate: https://github.com/tokio-rs/tracing
- wasmtime - A standalone runtime for WebAssembly: https://github.com/bytecodealliance/wasmtime

References To Other Episodes
- Rust in Production Season 1 Episode 1: InfluxData: https://corrode.dev/podcast/s01e01-influxdata

Official Links
- Arroyo Homepage: https://www.arroyo.dev/
- Arroyo Streaming Engine: https://github.com/ArroyoSystems/arroyo
- Blog Post: Rust Is The Best Language For Data Infra: https://www.arroyo.dev/blog/rust-for-data-infra
- Micah Wylde on LinkedIn: https://www.linkedin.com/in/wylde/
- Micah Wylde on GitHub: https://github.com/mwylde
- Micah Wylde's Personal Homepage: https://www.micahw.com/