Oxide and Friends When Async Attacks!
7 snips
Jun 27, 2025 
Dave Pacheco, a software engineer at Oxide, shares insights on connection pooling; Eliza Weisman, an expert in Rust, discusses the Tokio runtime; and Augustus Mayo reveals his findings on performance regressions in their Colo environment. They dive into the debugging labyrinth to uncover hidden delays in request handling and the complexities of multi-threaded task management. Their journey includes using D-Trace probes and visualization techniques, all aimed at optimizing system performance and tackling the challenges of asynchronous programming.
AI Snips
Chapters
Transcript
Episode notes
Latency Regression After Disk Removals
- The Oxide team faced a latency regression after an update involving disk removals causing database strain.
- Multiple people noticed errors around database access, sparking an immediate investigation led by Augustus Mayo.
Database Exonerated Early On
- Database queries showed acceptable response times, but HTTP requests still experienced long delays.
- This mismatch pointed away from the database as the root cause of the latency issue.
Async Tasks Obscure Debugging Context
- Rust async tasks complicate debugging because events happen in different tasks than the work being done.
- Traditional DTrace stack traces are less useful as async tasks switch threads frequently, hiding execution context.
