Sled Recovery and Update Challenges

Two years ago, the Oxide team encountered data corruption during a fairly simple network data transfer. The ensuing debugging sessions uncovered a truly bizarre bug involving CPU speculation! Bryan and Adam were joined by colleagues John and Rain to discuss the discovery and circuitous hunt to track down the bug.

In addition to Bryan Cantrill and Adam Leventhal, speakers included John Gallagher, and Rain Paharia.

Previous episodes mentioned:

• OxF s03e09 - Get You a State Machine for Great Good
• OxF s03e20 - Shipping the first Oxide rack: Tales from Manufacturing
• OxF s04e25 - RTO or GTFO
• OxF s02e38 - A Debugging Odyssey

Some of the topics we hit on, in the order that we hit them:

• The Update Framework
• Omicron Issue #3441 (Oxide Computer GitHub)
• Omicron Pull Request #3455 (Oxide Computer GitHub)
• stlouis Issue #454 (Oxide Computer GitHub)
• Changing psrset.out.txt (Oxide Computer)
• Commit 5ec2885322423c0cca0d006611b5c9ac94b0f588 (Oxide Computer)
• Omicron Pull Request #3560 (Oxide Computer GitHub)

If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!