Bryan and Adam are joined by a number of members of the Oxide networking team to talk about the networking software that drives the Oxide rack. It turns out that rack-scale networking is hard... and has enormous benefits!

We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from February 27th, 2023.

In addition to Bryan Cantrill and Adam Leventhal, speakers included Ryan Goodfellow, Levon Tarver, Ben Naecker, and Arjen Roodselaar.

Links

Here's (much of) the live chat from the show:

ahl https://github.com/oxidecomputer/oxide-and-friends/blob/master/2021_11_29.md
ahl That's the Sidecar switch episode
bcantrill https://p4.org/
admchl What does "at line rate" mean?
Riking Line rate = As fast as the packets could possibly come. 1Gbit, 10Gbit, 100Gbit, etc
admchl Do you need ASICs to hit that speed? I assume x86_64 is not going to be fast enough for these specialised operations?
levon Yes, the Tofino 2 is the ASIC
bcantrill You need ASICs
bnaecker Yes, you really can't do these kinds of operations on a general purpose CPU.
rng_drizzt Yeah, you need specialized silicon here.
JustinAzoff Right, also often across all ports at the same time in both direction. a 48 port 10gbps switch will have a line rate of 960gbps (10 ** 48 ** 2)
duckman So the advantage is being able to offload compute to the switch?
bnaecker Yes, and specifically that you can separate the data plane (operations on the packets) from the control plane (decisions about what operations to allow or make).
tahnok What's TCAM?
levon Ternary Content Addressable Memory
bnaecker https://en.wikipedia.org/wiki/Content-addressable_memory#Ternary_CAMs
ryaeng Sure beats logging into a number of Cisco switches and making changes at the console.
admchl This is my favourite episode in a long time, this is all really fascinating.
rng_drizzt the first Sidecar episode was nearly 1.5 years ago ü§Ø , right after we cut the first rev
levon That episode blew my mind
duckman This sounds like a big deal on the scale of ebpf
duckman Or bigger
bnaecker It is extremely useful for understanding the processing pipelines. As long as you only run single-packet integration tests üôÇ
od0 just want to go out and find things to write P4 code for
JustinAzoff <@354365572554948608> yeah one way to think about that sort of thing is that xdp can be used to run little programs on a nic, where p4 is kind of like that, but running on effectively a nic with 48+ ports
bcantrill https://github.com/oxidecomputer/p4
SyntheticGate sidecar is the "codename" of our switch box
SyntheticGate "gimlet" is our server sled
bcantrill https://github.com/oxidecomputer/propolis
wmf So you have P4 and OPTE in the hypervisor at the same time?
bnaecker OPTE is in the host kernel.
arjenroodselaar The P4 runtime Ry described only exists in the test bed, where it high level simulates the switches. OPTE is part of the production environment.
arjenroodselaar The rough difference between P4 and OPTE is that P4 works on individual packets without much concept of a session (so it can't reason about TCP streams, packet order etc, so no firewall like functionality), while OPTE aims to operate on streams of packets.
JustinAzoff So you can run 100 VMs on a test system and wire them up to your virtual switch compiled by x4c?
arjenroodselaar Correct.
bcantrill OPTE == Oxide Packet Transformation Engine
admchl Gimlet?
rng_drizzt Compute server
rng_drizzt The Sidecar switch is actually just a PCIe peripheral to a Gimlet.
bnaecker The Gimlet managing the Sidecar is often called a "Scrimlet" for "Sidecar attached Gimlet"
Riking and "how do i reconfigure this giant network without hosing my ability to reconfigure this giant network"
ShaunO can identify with that - we seriously struggle to keep our own products inter-operating, let alone anyone else's
levon It can feel like a Sisyphean task.
a172 Setup a much smaller/simpler network in parallel that is accessible from "not your network" that gets you to the management interface.
levon It's a whole new world when you can look at the actual table definitions in P4
rng_drizzt Owning all the layers here is immensely beneficial
levon Those DTrace probes have been very helpful
bnaecker Those probes turned out to be everywhere. They are are in: SQL queries, HTTP queries, log messages, Propolis hypervisor state, virtual storage system, networking protocol messages, the P4 emulator, and probably more that I'm forgetting about.
levon For those unfamiliar with the DTrace tool, or the rationale behind leveraging DTrace over other tracing / debugging tools: https://www.cs.princeton.edu/courses/archive/fall05/cos518/papers/dtrace.pdf
bcantrill https://github.com/oxidecomputer/progenitor
ahl some notes on rust codegen: https://github.com/ahl/codegen-template
arjenroodselaar DDM! Bring us home!
a172 it astonishes me how many "cloud" type architectures are built on v4 only or v4 first.
a172 IPv6 is older than Wi-Fi
a172 It solves real problems. PLEASE use it.
nyanotech yessss fina...

Euro-defence, euro-booze and euro-TV