Navigating the Debugging Labyrinth

What happens when the Oxide API is slow? A podcast episode! More specifically, one about how the team employed all manner of debugging techniques to track it down to one obscure and configurable async runtime feature! Bryan and Adam were joined by members of the team to talk about that journey and the tools we used (and made!) along the way.

In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues, Dave Pacheco, Eliza Weisman, and Augustus Mayo.

Previous episodes mentioned:

• Oxide and the Chamber of Mysteries
• The Saga of Sagas
• DTrace at 20
• Cultural Idiosyncrasies
• Mr. Nagle’s Wild Ride
• A Debugging Odyssey
• RTO or GTFO

Some of the topics we hit on, in the order that we hit them:

• Falling in Love with Rust
• Tokio Runtime Builder – disable_lifo_slot
• magic‑trace (GitHub)
• Magic Trace podcast episode from Jane Street
• diesel‑dtrace (GitHub)
• omicron issue comment
• qorb
• statemap
• tokio‑dtrace
• tokio issue #7411
• Visualizing Systems with Statemaps
• PostgreSQL WAL INIT ZERO
• Statemaps: Visualizing System Behavior (YouTube)

The statemaps that we referred to:

• Nexus by thread, discussed starting at 55:29. (This statemap has some states coalesced; the full version is also available.)
• Nexus by Tokio task, tagged by thread, discussed starting at 1:15:33

The D scripts that we referred to:

• nexus-statemap.d used to generate the initial statemap
• nexus-profile.d to understand what was consuming CPU
• tokio-statemap-tagged.d to generate the Tokio task statemap

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!