Did Percona build their own hardware across the globe? What's their stack? No,

we are a customer of Percona and we're actually using all of their Kubernetes, Percona operators and tooling. So what we actually ended up doing is shoring up our, we shipped Fly Kubernetes service, FKS. Someone on Twitter the other day was like, you should name it something other than FKS. Cause that sounds like, so I was like, no, that's, that's, that's on purpose is FKS for that reason. So we will keep it. Are you a Silicon Valley

fan by any chance?

Oh my God. Yes. Then

you know where I'm going with that, right? This guy.

Vaguely. Wait, you got to say it. You got to tell me cause I'm going to laugh again.

Jeez. I'm, I'm on the person's name. He's the VC. Gosh, what

is his name? Oh, the doors that go like this and not like this. Yes, that guy. It says this guy f***s a lot. That's right. That's right. Yes. That's not why we named it that, but it was not an accident that is FKS. Anyway, we shipped this. Kubernetes runs really well on top of fly machines. And we didn't have all the features in it we needed for bog-standard Kubernetes operators to work properly. And so the first, actually, what we did was we decided we're going to run out-of operators from Percona to start. We're not going to fork any of this. We're not going to solve problems from scratch. What we're going to do is make our kubernetes work well enough to use percona which is a was a it's a little counterintuitive but it made sense for us so like percona has a bunch of products for basically launching managing upgrading doing backups for all of the things you need to do with databases on top of kubernetes and we're kind of building around that and part of of the reason for this is we know we can go beyond that at some point, but we don't want to start with something like Postgres major version upgrades, for example, are like a nightmare to build. And there's zero reason for us to build this. It already exists. It's like that's obviously something we shouldn't spend time on. And so Percona, it's like a basically it's like a builder by vendor that we've decided to buy from, if that makes sense for our particular users. And I think that one of the neat things about what we're doing, because I'm in some ways let go of the novelty of it, is we don't need anything novel for Postgres. We need really reliable Postgres for our users because that's all they're asking us for. And so I don't have to come up with a VC pitch for how our managed Postgres going to be, you know, world beating because it's just not the purpose of this thing. It will be world beating, but it's only because it's not it's doing what people are asking for instead of what I'm pitching to VCs, if that makes sense.

Very interesting. Well, I'm excited for you that you've, you know, four years later, you've come back to where you began

and where we began.

I mean, we really came. That's not the only reason, but one of the many.

I had an advisor slash investor basically shit themselves and I told them we were going to do our own Postgres service because they're actually looking at what PlanetScale and Neon are doing. And these are what I'd call like exotic database services. They're actually like being, they're actually building like serious, serious engineering to like change how Postgres does storage or make Vess which is that's infinitely scalable version of my sequel like like develop the actual database engine themselves and i think that the neat the neat thing about like watching amazon on this is like we don't need those things we really just need rds which is like the vanilla version of this database it just works well and we can charge whatever we need to charge to make that happen so it's been it was kind of funny to like actually start looking at the at what investors see as the postgres market and compare it to what we actually need to do and how very different it happened to be so

yeah i think your use case is different obviously it doesn't have to be full-on neon because you're not trying to attract you know someone who would manage fleets necessarily maybe i suppose on top of you i mean maybe maybe i actually get back to the same problems to some degree like you still have serverless right it would still be managed it'll still be serverless what

i need is something that works well for people who spend 25 to 2500 a month on their database that's that's basically like that's it doesn't need to be less than that and it doesn't need to be cheaper than that so But yeah, maybe we get back to that. I think that you could run Neon on fly. I think this is the cool thing is that you can actually build the exotic stuff on top of us someday for your own customers. That's just not what our customers need.

What's stopping somebody from tigressing the Neon, so to speak, on fly? If you can build a Neon on fly well

that would be neon doing it and they're just not it's the the that's not it didn't it's not something that's going to work out it's like that's not how their company needs to go they didn't i think one of the things that happens with companies is they launch with like neon raised so much money they launched with huge expectations and we don't like you don't your ego is necessarily big if your expectations are big does that make sense like you look far forward and be like i'm not going to give up that part of the market even though it may not be relevant at that time it's like no we have big expectations with money we're going to do big things and so uh it's neon not building neon on fly made sense but that would be the company that did it. There's not really another one. There's been a couple of small ones.

Have you heard the conversations of us talking about our CDN saga? Have you paid attention at all? No. I'm going to do my best because I'm blessing the details. Gerhard and Jared are deep in these CDN saga issues. And I'm going to try my best to not be negative, but we've not had the best experience with our CDN. It's been challenging. We've had some challenges and they seem insurmountable. And so we essentially came back to, well, we really, we need a simpler version. It's almost like what you just said with, we don't need, speaking for you, we don't need to be a PlanetScale or a Neon. We just need this RDS, this sort of like simple or sliver of Postgres, whereas the same for us. People go to CDNs as a media company and have infinite needs. Right. And we don't have those infinite needs. We have very simplistic needs, but we still need the kind of crux of what a CDN is for our little indie media company. And we have this thing called Pipe Dream. We'll talk about it next Friday. Actually, this Friday. Sorry, this Friday. Officially, and then it'll release the following Friday on our podcast. So it's December 4th, listeners. Kurt, December 4th, you know, this is not, not Friday, the seventh or sixth. Yeah. Sixth. It's like Friday, whatever next week. I'm not in front of a calendar. I'm trying to do math in my brain on 13th.

We'll go with 13th Friday, the 13th way actually is too. Good luck with that okay could be a reason

for that who knows so our idea is let's build a really simple cdn for us on top of fly and so that's what we're currently doing we're it's not i'm not sure if it will be the future yeah but for a while it's been an experiment a toy let's let's see if we can actually do it doesn't make sense can we solve our own problems can we build this little thing on fly and the reason i bring that up is because i said well could you tigress neon on fly and well maybe we don't need to be a neon on fly maybe we just need to be our own version of our own cdn on fly and it's our own we never had to go and build all these servers you build. We never had to go and, you know, globally distribute CPU and compute like you've done. We could just leverage the fact you've done it on our own. And maybe potentially it could be something that's usable by other people because it's just really simple. It's everything Cloudflare is, everything Fastly is and others that are like them, but just the simplistic version of it, the varnish layer, the simplistic varnish layer, not the complex crazy crap.

Yeah, okay, that's actually really exciting to me because this entire company exists because I was annoyed that there was no cloud I could build a CDN on top of, if that makes sense. As an individual developer, I could not ship a CDN because Fly didn't exist effectively. And so actually, actually I was really excited when we launched Tigris because to me that was like the last bit of the puzzle I would have needed to build a CDN. And so I'm fascinated you all are doing this because like, I love that you say simplest because like CDNs do a lot of like really interesting stuff that you may not need, but like really at the core of it, you just put a file somewhere and they make sure it's fast for other people in other places, which is all like cramming something in Tigris does now. It's just like, yeah, it's just there. You don't have to. And even Fastly at the time, I remember when Fastly got big because you were talking about DigitalOcean with SSDs and Fastly took off because they were a cdn with instant purge it was like that simple it was like they went to all the media companies and said when you ship a typo that's really embarrassing you can purge it and nobody else will ever see it like within seconds it's not going to be there for another few hours which is what was happening with akamai and others at the time and so uh it's it's kind of funny because like the infrastructure will now support instant purge you don't need a cdn to like build a bunch of shit for you to do that anymore you just need to use Tigris effectively and then have a button that deletes an object from Tigris and you're done. Or just uploads a new one. Anyway, that's really cool. I'd love to hear more. What stack are you building this with? Is this a lecture?

So I'm not building the application. Gerhard is building it for the most part i think jared's chiming in on different details it's being built on varnish oh

nice so

we're not using tigris at all oh

that's interesting you should have them look at tigris and see if that changes how they build it because i feel like varnish is even harder than i think it needs to be maybe

it's possible i think it would be a good conversation with Garrett. He's definitely in the details of the what's and the why's. I am more like, you know, let's just figure it out kind of thing. Because we've been, this is in the weeds a tiny little bit, but we've been bottlenecked by our inability to move to the next thing when it comes to a CDN. And much love to Fastly. They've been amazing to work with over the years, but there's challenges with, there's a lot of challenges. I think in particular, like the VCL inside of Fastly, just to be very specific, is not versioned.

Right.

So, you know, Gerhard and Jared, two developers on our team, I would chime in too, but I would just like, that would just ruin it. It would be the worst. They have to coordinate like humans would coordinate like, Hey, I'm working on the VCL right now. Don't touch it. Or this is the version of it and export it by copy and paste into our own Git repository. So it's shadowed by version. It's like even something that simple, like that's innovation the fantasy layer that we would absolutely love yeah but it's just not there or apis changing and things break for us we're like why is our our fees not updating why are these things happening oh yeah the api changed and we were not made aware of this api change like i think it's kind of prudent to tell a developer when your api

changes yes

i mean and maybe they did and maybe they did. I don't know. Right.

That's actually a hard problem is communicating this stuff to people is actually incredibly difficult, even if you decide to do it.

So I'm not trying to say they're bad. I'm just saying we've had some hurdles over years, you know, over years with this. And they're aware of it. And they may even be listening right now. So I'm really sorry we had to bring this up, but it's just... Oh,

somebody there knows. Yeah.

I mean, they pay attention to any time we talk about Fastly for some reason, shape or form. It happens. Fastly

and Cloudflare are fascinating to me because in some ways they're doing things on hard mode because they got big and then had the money to do things in a way that wouldn't make any sense to people like us. So we originally started four iterations ago, this was pre-2020. We shipped a multi-tenant JavaScript engine for just running JavaScript at the edge, which is just like Cloudflare workers. And then that didn't do what we wanted. I got big companies were happy to come buy this and build stuff on top of it, but I wanted individual devs to just ship apps. And individual devs were not interested in writing JavaScript some unknown platform. And I was always grateful that we didn't have the captive customer audience that Cloudflare does. Because Cloudflare has so many customers, they ship multi-tenant JavaScript, they get enough traction from it, they think it's successful. And the reality is, those customers were just willing to use more of Cloudflare's features. It wasn't like attracting new customers at the rate that we needed to as a startup, for example. And Fastly is really similar because they did this instant purchasing. It was all based on Varnish. And when you kind of start with this black box, how you evolve from there is really kind of hard. And we got really lucky because at some point, and I don't want to say this is because I'm smart. It's because it wasn't working. My brain flipped from like, I want to build a CDN that devs can take advantage of, which is where you'd get scripting CDNs and customizable VCL and all of these things, into why are we building proprietary stuff for the CDN when in theory you could just have a cloud that lets you run a CDN on it pretty easily. I'm really happy we flipped for that reason, because I think that there's probably people within both those companies that understand how constrained their path has been. And it's all because they were successful. I'm not going to say at the wrong time, it's because they were wildly successful and like locked them into this decision that doesn't make any sense if you're starting from scratch anymore. And I think VCL is the version of that. And then they also, when Fastly did the Wasm stuff, it was the same way. I was like, wow, this looks like a project a company does when it has too much money to spend and isn't being forced to be pragmatic about how people use the thing. So anyway, it's just really interesting you all are experiencing this because I've watched specifically those two companies for like eight years at this point. And I've watched them kind of be at the mercy of their previous choices in a way that like cloud provider infrastructure hasn't been. And it's quite the same way. Let's

time box this, what I'm about to share, to five minutes or less.

That's a good code for talk less, Kurt. Be more efficient. I love

it. No, no, no. Not at all. I have bigger things I want to talk to you about, but I'm really, I just put, this is all open source. So I just shared a URL with you here in Riverside. So go to that URL and just peruse briefly the code base because it's very small. Just give me a glimpse and an initial reaction. This

is, I wrote, one of my favorite things I wrote for our blog was called the five hour CDN. This

inspired it.

Yeah. Okay. Yeah. I love it.

Yeah. We actually, so Jared is quoted here and on a podcast, we do these shows called Kaizen, which do you know the word Kaizen? No. It's Japanese for continuous improvement or always be improving. And so we've of many pillars, give them what they came for, keep the main thing, the main thing, slow down and check yourself when you're going too fast and Kaizen. Like these are the four pillars of our psyche when it comes to our business. And so Jared on a podcast, on a Kaizen podcast, where we're introspecting what we're doing and I've shared with you our challenges. And now with the rest of the podcast world that's listening, Jared said, I like the idea of having this like 20 line varnish config that we deploy around the world. And it's like, look at our CDN guys. And that's it. Like, and so that's what he said in the podcast. And so Gerhard Lazu, our resident SRE and friend for many, many years now here at ChangeLog, a prior host of Ship It, the podcast, etc. He's still involved in all the things we do. He planted that seed in his brain and went away and over time brought out this idea of this pipe dream. Jared called it a pipe dream. Could we actually do this? Yep. Could we build our own CDN on fly? And your blog post was fodder for the possibility and enablement, so to speak. Yep. so we said, well, is that even possible? Should we even do it? And I was like, no, because I want to work with a partner that does it. I don't want to manage more code. I don't want to be responsible for our CDN. So here's me thinking as a businessman around this 80 media company, no, we should partner and get them to pay us because that's how it should work. We should choose a major winner, enable a symbiotic relationship and share our story with the world through how that works out. That may not be how this ends up working out. So this is still a pipe dream. That's why we call it pipe dream because we're not sure if it will work out, but this is it. A single purpose multi-tenness EDn for just us that runs varnish cash it's open source it runs on fly and that's that's where we're at so far oh

that's really cool that's uh you've got me like thinking i should go spend the weekend and and just do this from scratch and see what i'd come up with because like since i wrote that blog post tigers exists which is i think I might do it different now, but maybe I'm wrong. Yeah. I apparently need my, I cope with burnout by spending a week writing a little demo. So my last one was called BFAS. It's Bash Functions as a Service. And it was a, it hooked up to the ChatGPT API. And I said, write me some Bash and then it would run it in a machine. And I was actually using it to see if I could hack our own machines. And ChatGPT is really good at writing exploits. So that was kind of fun. So that's maybe my next one.

Interesting. So, I mean, in light of this and having one and a half minutes left on this brief part of the conversation, is we've considered, well, if this does make sense for us, who else needs a really simple CDN? Yeah. That's not so much anti-fastly or anti-cloud but it's just bloated. We don't need all those features. We just really need something simple. Maybe we could flesh out what is PipeDream into a Tigris and be part of this alliance.

So we've all seen how my predictions go with Rebel Alliance, but I have this little bit of a hot take that like in the future like cdn features will probably just be part of your app not necessarily a whole separate service which is a my take on that is i think that now like you get to cdn it's actually expansive because it's everything from like ddos protection to bot abuse protection to like optimizing video streams as they flow through and all kinds of stuff like that. But for actually just storing kind of chunks of files and getting them back to users pretty fast, I feel like that's just the thing apps will do. It won't be like an entirely separate service anymore. And so I'm very curious what your exploration finds for this. And I actually would love to ask them, I'd love to talk to them and be like, what, like, I'm curious what the benefit of a separate service is from your existing Phoenix app for something like this. Like, why is it, why does it make sense to have a CDN as a bolt-on and not have the Phoenix app just be doing this work? We

should do a podcast about that. We should. We should invite you on a Kaizen and talk, you know, in depth about the possibility.

I've been meaning to do a Twitch stream of building something, and I actually wonder if the dumbest CDN ever on a Twitch stream would be a fun thing to build. My version, Kurt's version, like Taylor's version of the Taylor Swift songs. Sure,

sure, sure. I

really love this, though. I actually like, this is the exact type of thing that I love seeing on fly.