Integrating Rust and Python for Data Processing

Bytewax is a curious stream processing tool that blends a Python surface with a Rust core to produce something that’s in a similar vein to Kafka Streams or Apache Flink, but with a fundamentally different implementation. This week we’re going to take a look at what it does, how it works in theory, and how the marriage of Python and Rust works in practice…

–

The original Naiad Paper: https://dl.acm.org/doi/10.1145/2517349.2522738

Timely Dataflow: https://github.com/TimelyDataflow/timely-dataflow

Bytewax the Library: https://github.com/bytewax/bytewax

Bytewax the Service: https://bytewax.io/

PyO3, for calling Rust from Python: https://pyo3.rs/v0.21.2/

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

Kris on Twitter: https://twitter.com/krisajenkins

--

#softwaredevelopment #dataengineering #apachekafka #timelydataflow