Is Gadsby the Data Layer?

The following is my conversation with Abhi Aiyer and Ward Peeters, two lead engineers behind Gatsby Cloud and the recently announced Gatsby v4, which is at the forefront of what I think is the most significant change in the Jamstack landscape in the past 2 years.

Watch the video version here.

Links:

• Gatsby 4
• Netlify DPR
• My blogpost on Smart Clients vs Smart Servers

Timestamps: 

• [00:00:00] Cold Open 
• [00:00:28] Swyx Intro 
• [00:01:59] Call Start 
• [00:03:07] Gatsby v4 
• [00:06:23] Incremental Builds 
• [00:07:16] Cache Invalidation 
• [00:09:03] Gatsby DSG vs Netlify DPR 
• [00:09:35] Abandoning Redux for LMDB 
• [00:11:50] Parallel Queries (PQR) 
• [00:13:32] Gatsby DSG 
• [00:15:24] Netlify DPR vs Gatsby DSG 
• [00:19:19] The End of Jamstack 
• [00:22:12] Tradeoffs and Performance 
• [00:24:34] Image Processing 
• [00:27:25] Automatic DSG 
• [00:29:33] Gatsby Cloud vs Netlify 
• [00:33:34] Gatsby vs Next.js 
• [00:35:41] Gatsby and the Content Mesh 
• [00:37:19] React 18 and Gatsby 
• [00:39:45] Custom rendering page fragments with React 18 
• [00:42:10] Server Components in Limbo 
• [00:43:33] Smart Servers vs Smart Clients 
• [00:45:21] Apollo and Open Source Startup Strategy 
• [00:47:06] TMA: Too Many Acronyms 
• [00:49:16] Gatsby for Docs 

Transcript
 

[00:00:00] Cold Open 
 

[00:00:00] Abhi Aiyer: And so with LMDB in place, right? We have workers that can read and write to LMDB, which allows us to run parallel queries. So PQR was a huge advancement for us. I think we saw up to like 40% reduction in query running time. And build times went down. We had a goal, I think it was like, we'd try to look for at least 20% reduction in build times and I think we hit 26%, so all cool wins, you know? 

[00:00:28] Swyx Intro 
 

[00:00:28] swyx: The following is my conversation with Abhi Aiyer, and Ward Peeters, two lead engineers behind Gatsby Cloud, and the recently announced Gatsby V4, which is at the forefront of what I think is the most significant change in the JAMstack landscape in the past two years. We discussed how parallel query writing PQR and deferred static generation DSG are achieving 40% faster queries and 300% faster overall builds. 

[00:00:53] And they did a wonderful job handling the most impolite questions I could think of, including whether it Gatsby Cloud is a Netlify clone or the Gatsby should just be a data layer on top of Next.js and how they're dealing with TMA too many acronyms in web development. This conversation should be viewed together with my past discussions, with Sunil Pai and Misko Hevery in considering the cutting-edge of web development today. Online discussions often present a binary split in that your technical choices either have to optimize for developer experience or user experience. 

[00:01:25] But I find that it is builders like Abhi and Ward and Misko and Sunil who are constantly trying to improve the experience of developers in building great user experiences by default. I hope you enjoy these long form conversations I'm trying to produce with amazing developers. I still don't have a name for it. 

[00:01:41] And I still don't know what the plan is. I just know that I really enjoy it. And the feedback from you guys have been really great. So if you like this, share with a friend, if you have other requests for guests, tag them on social media, I basically like to make this a space where passionate builders and doers can talk about their craft and where things are going. 

[00:01:58] So here's the interview.  

[00:01:59] Call Start 
 

[00:01:59] Abhi Aiyer: I'm Abhi Aiyer. I'm a principal engineer at Gatsby. Thanks for having us.  

[00:02:05] Ward Peeters: My name is Ward Peeters. I'm a staff software engineer at Gatsby and I'm from Belgium. And I've been working mostly on the open source side.  

[00:02:15] Abhi Aiyer: I forgot to say where I'm from. I'm from Los Angeles, you know, Hollywood,  

[00:02:21] swyx: I'm actually heading down to LA,  

[00:02:22] Abhi Aiyer: in a couple of weeks, there's,  

[00:02:24] swyx: I'm going to Kubecon, which is like a very interesting thing for a front end engineer to end up at. But that's where my career has taken me.  

[00:02:34] So this conversation started because I had a chat with Sunil, on this podcast that I accidentally launched. I don't think we did Gatsby much, a good favor. 

[00:02:45] Like we both saw the new updates and I didn't get to say the nice things that I thought about Gatsby. I should also say that I used to have my blog on Gatsby and I no longer do. I used to work at Netlify and I no longer do. There's a lot of history here for me with Gatsby. It's been a while since I caught up, and I'm curious to see or get the latest. 

[00:03:07] Gatsby v4 
 

[00:03:07] swyx: Maybe we should start off with like a quick summary of what's new with Gatsby with Gatsby V4, right?  

[00:03:13] Abhi Aiyer: Is that a good place to start? Yeah, I think so. 

[00:03:17] swyx: So first of all, I think the marketing was really nice. Gatsby camp, it seems like a really big push and qualitatively very different from Gatsby 3. Tell me about what the behind the scenes was like. 

[00:03:30] Abhi Aiyer: Yeah, it was, we're getting better at the marketing side of what we're doing these days and Gatsby 4 was a big push. It really changed how we approach the framework as a whole.  

[00:03:43] For those who don't know, traditionally Gatsby was a static site generator, purely static. We hold ourselves high on our connections to a content management system. 

[00:03:55] And we provide a really good data layer there, that takes all those requests that you would normally make to a content manager system, turns them into a, like a store of data that you can then use and query from graph QL. And the big thing that we were hitting before gas before was. Company was growing. 

[00:04:17] And as more customers were using Gatsby cloud, we started realizing that we couldn't scale to really large sites and large sites is like a misnomer. Like you could be, you could be a 50,000 page site and be considered large given the data that you may have. But we're talking like hundreds of thousands of pages. 

[00:04:38] And the thing that we kind of realized is not all pages are created equal on your site. Especially the ones from like 20, 15, 20 14, where, you know, no one's looking at that people, those pieces of content, if you're a site with a huge archive o...