Oxide and Friends The Network Behind the Network
May 9, 2023
01:39:08
Rejecting Commodity BMCs
- Oxide rejected traditional BMCs because of the complexity and security risks observed in commodity machines.
- They created a stripped-down microcontroller called the service processor running Hubris and Humility to control the management network reliably.
NCSI Challenges Expose Risks
- NCSI promised reduced cabling by multiplexing management and host traffic on the same link, which was very attractive.
- However, implementation issues, firmware reliability, and unclear reset guarantees made NCSI an unviable solution.
Cabled Backplane Changed Game
- Moving to a cabled backplane with blind-mated sleds eliminated the original cabling reduction benefit of NCSI.
- This, combined with uncertainty about NCSI, led to adopting a dedicated management network instead.
Get the Snipd Podcast app to discover more snips from this episode
Get the app 1 chevron_right 2 chevron_right 3 chevron_right 4 chevron_right 5 chevron_right 6 chevron_right 7 chevron_right 8 chevron_right 9 chevron_right 10 chevron_right 11 chevron_right 12 chevron_right 13 chevron_right 14 chevron_right 15 chevron_right 16 chevron_right 17 chevron_right 18 chevron_right 19 chevron_right 20 chevron_right 21 chevron_right 22 chevron_right 23 chevron_right 24 chevron_right 25 chevron_right 26 chevron_right 27 chevron_right 28 chevron_right 29 chevron_right 30 chevron_right 31 chevron_right 32 chevron_right 33 chevron_right 34 chevron_right 35 chevron_right 36 chevron_right 37 chevron_right 38 chevron_right 39 chevron_right 40 chevron_right 41 chevron_right 42 chevron_right 43 chevron_right 44 chevron_right 45 chevron_right 46 chevron_right 47 chevron_right
Introduction
00:00 • 3min
How to Search Hacker News for Hubris and Humility
02:38 • 2min
How to Connect Your Service Processors to the Internet
04:32 • 2min
The Future of Network Channel Side Band Interconnect
06:14 • 2min
How to Reset a Nick in NCSI
08:00 • 3min
The Problem With RTL Implementations
11:21 • 2min
The Power Domain Concerns of a Server
13:00 • 3min
The Importance of a Separate Management Network
15:42 • 2min
The Origin Story of Monorail
17:39 • 2min
Monorail of the Key
19:53 • 4min
How to Build a Gigabit Ethernet Switch
23:30 • 6min
How to Make a Monorail Link That Works
29:03 • 2min
The Origins of Monorail
30:50 • 1min
The Monorail Problem
32:17 • 2min
How to Connect a Microcontroller to an Ethernet Switch Using an RMI Interface
33:58 • 5min
The Impossibility of Finding a Standard for a Hundred Base Effects
38:43 • 2min
The Def Kit for the VSE 7448
40:24 • 2min
How to Install a MIPS Core on a Dev Board
42:05 • 2min
The Microchip Ethernet Switch Api Is Open Source
44:27 • 2min
How to Configure a Soft Emulator for a Service Processor
46:00 • 2min
The Problem With Yanging Up Groups of Four Surveys
47:38 • 2min
How to Tune a QSDMI Link Remotely
49:15 • 2min
The Value of Disdain
50:49 • 1min
The Mystery of the Firmware Payload
52:19 • 2min
The Importance of Using Indirect Rights in a Chip
53:58 • 1min
How to Reverse Engineer a Microchip for 100 Giggling
55:23 • 2min
The Importance of Open Source Drivers in Automotive Applications
57:11 • 2min
How to Build a High-End Part
59:02 • 2min
The World's Worst Doxigen Parser
01:00:55 • 2min
How to Configure a Switch on Your Own
01:02:45 • 3min
The Importance of Monorail Status
01:05:30 • 2min
How to Fix a Broken Link on a 7448
01:07:10 • 2min
How to Debug Gamelits
01:09:11 • 2min
The Better Structured Hacky Tooling Is Funnier
01:10:59 • 2min
The Management Gateway Service at Oxygen
01:12:53 • 3min
Humility Console Proxy
01:15:31 • 3min
The Importance of Humility Console Proxy
01:18:03 • 1min
How to Debug a Service Processor
01:19:30 • 2min
The Coolest Thing We Built So Far With the Management Network
01:21:10 • 2min
How to Stream an OS Image to a Machine That Has No Hard Drives at All
01:23:26 • 2min
How to Program a M.2s
01:25:49 • 2min
How to Reprogram a Machine in Parallel
01:27:24 • 2min
The Crucial Path to a Successful Recovery Process
01:28:55 • 3min
The Importance of Using a Console Proxy
01:31:37 • 2min
The Importance of Getting the System to Work
01:33:24 • 2min
The Importance of Ignition Control in Hubris
01:35:07 • 2min
The Importance of Humility in Open Source
01:36:53 • 2min
Bryan and Adam are joined by Oxide colleagues Arjen, Matt, John, and Nathaneal to talk about the management network--the brainstem of the Oxide Rack. Just as it ties together so many components, this episode ties together many many (many!) topics we've discussed in other episodes.
We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from May 8th 2023.
In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues Arjen Roodselaar, Matt Keeter, John Gallagher, and Nathanael Huffman.
This built on work described in many previous episodes:
- Cabling the Backplane Prior to going all-in on a cabled backplane with blind-mated server sleds (i.e. no plugging, unplugging, mis-plugging network cables). We (Bryan) espoused an "NC-SI or bust" mantra... at least in part to avoid doubling the cable count. With the cabled backplane, the reasons for NC-SI disappeared (which let the many reasons against truly shine).
- The Pragmatism of Hubris in which we talk about our embedded operating system, Hubris (and it's companion debugger, Humility). Hubris runs on the service processors that are the main endpoints on the management network. Matt's work controlling the management network switch (the VSC7448) is in the context of Hubris, as is John's work communicating with the sleds over the management network.
- The Power of Proto Boards showed and told about the many small boards we've used in development. Several of those were purpose built for controlling and simulating parts of the management network.
- The Oxide Supply Chain Kate Hicks joined us to talk about the challenges of navigating the supply chain. Mentioned here in the context of "supply-chain-driven design": we designed around the parts we could procure! Tip: stay away from "automotive-quality" parts when the auto industry is soaking them all up.
- Holistic Boot in which we talked about how (uniquely!) Oxide boots from nothing to its operating system and services. Over the management network, we can drive server recovery by piping in a RAMdisk over the network and then (slowly) through the UART to the CPU.
- Get You a State Machine for Great Good Andrew joined us to talk about his work on a state-machine driven text-UI and its companion replay debugger. We mentioned this in the context of John replaying the long upload process in seconds rather than hours to fix a UI bug.
Major components of the management network
Matt's VSC7448 dev kit
Matt's remote tuning setup via webcam
Management network debugging
Management network debugging