The InfoQ Podcast Building a Reliable Kafka Data Processing Pipeline With Lily Mara
10 snips
Aug 14, 2023 AI Snips
Chapters
Transcript
Episode notes
OneSignal's Initial Architecture
- OneSignal initially used a Rails monolith for all operations, including HTTP handling and asynchronous job processing with Sidekick.
- This setup worked well for sending large message campaigns but encountered scaling challenges with high-throughput HTTP endpoints.
Introduce Queuing for High Throughput
- Implement queuing to control concurrency and load on Postgres when dealing with high-throughput HTTP endpoints.
- OneSignal achieved this by switching from synchronous processing in HTTP handlers to a Kafka producer and a Rust consumer.
Early Adoption of Rust
- OneSignal adopted Rust early on for core components like OnePush, valuing its speed, low overhead, and ease of development.
- They were early adopters, using Rust shortly after its 1.0 release, and have been pleased with its performance and maintainability.