The Stigma of Rack Scale Design

On this week's show, Adam Leventhal posed questions from Hacker News (mostly) to Oxide founders Bryan Cantrill and Steve Tuck. Stick around until the end to hear about the hardest parts of building Oxide--great, surprising answers from both Bryan and Steve.

They were also joined by Steve Klabnik.

Questions for Steve and Bryan:

[@6:38] Q:

Congrats to the team, but after hearing about Oxide for literal years since the beginning of the company and repeatedly reading different iterations of their landing page, I still don't know what their product actually is. It's a hypervisor host? Maybe? So I can host VMs on it? And a network switch? So I can....switch stuff? (*)

A:

Steve: A rack-scale computer; "A product that allows the rest of the market that runs on-premises IT access to cloud computing."Bryan: agrees

[@8:46] Q:

It's like an on prem AWS for devs. I don't understand the use case but the hardware is cool. (*)I didn’t understand the business opportunity of Oxide at all. Didn’t make sense to me.However if they’re aiming at the companies parachuting out of the cloud back to data centers and on prem then it makes a lot of sense.It’s possible that the price comparison is not with comparable computing devices, but simply with the 9 cents per gigabyte egress fee from major clouds. (*)

A:

Bryan: "Elastic infrastructure is great and shouldn't be cloistered to the public cloud"; Good reasons to run on-prem: compliance, security, risk management, latency, economics; "Once you get to a certain size, it really makes sense to own"Steve: As more things move onto the internet, need for on-prem is going to grow; you should have the freedom to own

[@13:31] Q:

Somebody help me understand the business value. All the tech is cool but I don't get the business model, it seems deeply impractical.

• You buy your own servers instead of renting, which is what most people are doing now. They argue there's a case for this, but it seems like a shrinking market. Everything has gone cloud.
• Even if there are lots of people who want to leave the cloud, all their data is there. That's how they get you -- it costs nothing to bring data in and a lot to transfer it out. So high cost to switch.
• AWS and others provide tons of other services in their clouds, which if you depend on you'll have to build out on top of Oxide. So even higher cost to switch.
• Even though you bought your own servers, you still have to run everything inside VMs, which introduce the sort of issues you would hope to avoid by buying your own servers! Why is this? Because they're building everything on Illumos (Solaris) which is for all practical purposes is dead outside Oxide and delivering questionable value here.
• Based on blogs/twitter/mastodon they have put a lot of effort into perfecting these weird EE side quests, but they're not making real new hardware (no new CPU, no new fabric, etc). I am skeptical any customers will notice or care and would have not noticed had they used off the shelf hardware/power setups.

So you have to be this ultra-bizarre customer, somebody who wants their own servers, but doesn't mind VMs, doesn't need to migrate out of the cloud but wants this instead of whatever hardware they manage themselves now, who will buy a rack at a time, who doesn't need any custom hardware, and is willing to put up with whatever off-the-beaten path difficulties are going to occur because of the custom stuff they've done that's AFAICT is very low value for the customer. Who is this? Even the poster child for needing on prem, the CIA is on AWS now.I don't get it, it just seems like a bunch of geeks playing with VC money?(*)

A:

Bryan: "EE side quests" rant; you can't build robust, elastic infrastructure on commodity hardware at scale; "The minimum viable product is really, really big"; Example: monitoring fan power draw, tweaking reference desgins doesn't cut it Example: eliminating redundant AC power suppliesSteve: "Feels like I’m dealing with my divorced parents" post

[@32:24] Q (Chat):

It would be nice to see what this thing is like before having to write a big checkSteve: We are striving to have lab infrastructure available for test drives

[@32:56] Q (Chat):

I want to know about shipping insurance, logistics, who does the install, ...Bryan: "Next week we'll be joined by the operations team" we want to have an indepth conversation about those topics

[@34:40] Q:

Seems like Oxide is aiming to be the Apple of the enterprise hardware (which isn't too surprising given the background of the people involved - Sun used to be something like that as were other fully-integrated providers, though granted that Sun didn't write Unix from scratch). Almost like coming to a full circle from the days where the hardware and the software was all done in an integrated fashion before Linux turned-up and started to run on your toaster. (*)

A:

Bryan: We find things to emulate in both Apple and Sun, e.g., integrated hard- and software; AS/400Steve: "It's not hardware and software together for integration sake", it's required to deliver what the customer wants; "You can't control that experience when you only do half the equation"

[@42:38] Q:

I truly and honestly hope you succeed. I know for certain that the market for on-prem will remain large for certain sectors for the forseeable future. However. The kind of customer who spends this type of money can be conservative. They already have to go with on an unknown vendor, and rely on unknown hardware. Then they end up with a hypervisor virtually no one else in the same market segment uses.Would you say that KVM or ESXi would be an easier or harder sell here?Innovation budget can be a useful concept. And I'm afraid it's being stretched a lot. (*)

A:

Bryan: We can deliver more value with our own hypervisor; we've had a lot of experience in that domain from Joyent. There are a lot of reasons that VMware et al. are not popular with their own customers; Intel vs. AMDSteve: "We think it's super important that we're very transparent with what we're building"

[@56:05] Q:

what is the interface I get when I turn this $$$ computer on? What is the zero to first value when I buy this hardware? (*)

A:

Steve: "You roll the rack in, you have to give it power, and you have give it networking [...] and you are then off on starting the software experience"; Large pool of infrastructure reosources for customers/devs/SREs/... in a day or less; Similar experience to public cloud providers

[@01:02:06] Q:

One of my concerns when buying a complete so...