Spencer Kimball Pt 1: Consistent Synchronous Replication

I enjoyed learning about how Cockroach is leaderless and synchronous and that makes it a lot more resilient than alternatives.

Audio source: https://changelog.com/founderstalk/75

Spanner paper: https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf

Share/Comment via Tweet: https://twitter.com/swyx/status/1376470276918050818

Adam Stacoviak: [00:00:00] So when did you encounter the problem that you're solving today? 

Spencer Kimball: [00:00:04] Yeah, so databases it turns out that they have been extraordinarily central in my career. Back as early as the.com startup. I did, we go systems we built sharded, Oracle and sharded Postgres as the two sort of flavors we supported. And I got to tell you when I was at Berkeley, I wasn't very interested in databases.

I mentioned graphics. That was really. Probably my key interest databases. I didn't take until my first and only year of grad school. And I just kinda took it to get some credits. I ended up being pretty interested in the course, but I didn't really think they'd be central to my career. But as soon as I hit the quote, unquote, real world databases became a central problem of big source of frustration at Wego.

And then when I got to Google, that was one of the first projects that got thrown onto, which was the AdWords system, which you know, was nascent then in 2002. But it was running into problems with sharded, my SQL. And you hear this word, "sharded", but it really, you know, for listeners that aren't aware of what that implies it's, it's about taking a monolithic database like Postgres or my SQL or Oracle that really is meant for you know, a single machine.

Even if that machine can be quite large and you say, well, maybe this is going to be large enough. And this is the case of ad-words when I got put on that project. So you'd say, okay, we're going to use two databases. I'll put half our customers on the first database, half on the second. And maybe at some point you start reaching capacity on those two.

And so then you say, we're gonna use four, we're going to use five, or we're going to use it got up to about 32. I think when I was at that project at Google and all these different problems started to occur as you started, you know, the application complexity became quite high. Just one ridiculous practical example:

the, my SQL databases had too many connections coming into them and you know, that started to cause them to cavitate. And so we solve these problems. Every morning, we had this ads war room to solve the latest set of problems related to this just basically scalability challenge with the database.

And you know, I would just say that in Google AdWords by the time they replaced that sharded my sequel architecture, they'd gotten to a thousand shards. So it became, you know, thousand my SQL instances. And I've heard that Facebook has hundreds of thousands of my SQL instances. So there's kind of no end to both how scalable that architecture is.

But also how much time you have to put in to truly keep scaling it. So that's, that's a scalability challenge. There's also resilience challenges.  And that is you really don't want to have a database that has a primary and a secondary, and that's been the standard way to operate databases for, you know, most of my lifetime.

The problem with that solution is that the secondary. He's getting an asynchronous replication stream of data. And even if you put it in another data center, so you have a really nice failure scenario, so you can lose a data center and fail over that fail over might imply data loss because that asynchronous replication stream might not have fully made it over to the secondary when the primary dies.

So you've switched over to the secondary and you realize, Oh, wait a second. I thought I had just sent that email out as an example, but it's not in my outbox. What happened? Well, the replication stream just didn't get that email into the outbox on the secondary. So it's almost like you've moved backwards in time.

You've regressed to an earlier version of the state that you had in an application. And that causes huge headaches, right? I mean, if a data center was lost at Google back in 2004, let's say it would be many teams scrambling to figure out what might've gone wrong. You know, did we charge a customer twice?

You know, are there. Consistency problems in the data. Cause some of the stuff got replicated and some other stuff didn't and you'd have to write cleaners and scripts that would go through things. And you just try to reason through what might've gone wrong with your use case that that's not the right way to do database replication.

And certainly not. In 2020 Google started to play around with better ways to do that. As early as 2004, 2006, they built big table. Then they built in called mega store and then they built something called Spanner. And Spanner is really what inspired cockroach. And so there's scalability, there's resilience.

Those are two of the biggest problems that I've faced with databases in my career that sort of gold standard these days with databases is to do what's called consistent synchronous based replication. The popular ways to do this is something called Paxos. There's something called raft and that what they do is consensus.

So instead of just writing to a primary and asynchronously replicating to a secondary, you actually write to. Three data centers or three replication sites, and you are going to be committed if the majority of the replication sites respond positively or affirmatively to any particular, right? If, for example, you only write to one out of three data centers that write.

Can't be committed. So you need two out of the three. And so as long as you always have two out of three, if you lose any one data center out of those three, you always are guaranteed that one of the remaining two has the exact data that you need. So as long as you only lose the minority, you have total operational sort of continuity.