The Swyx Mixtape

Listen to Code Story: https://codestory.co/podcast/bonus-adam-robinson-getemails/https://getemails.com/data-privacy/https://vimeo.com/48104974https://www.bizjournals.com/boston/blog/techflash/2013/05/former-ratepoint-customers-launch.htmlhttps://www.amazon.com/Permission-marketing-fastest-growing-companies-permission-ebook/dp/B087KTYCH1

The following is my conversation with Abhi Aiyer and Ward Peeters, two lead engineers behind Gatsby Cloud and the recently announced Gatsby v4, which is at the forefront of what I think is the most significant change in the Jamstack landscape in the past 2 years.Watch the video version here. Links:Gatsby 4Netlify DPRMy blogpost on Smart Clients vs Smart ServersTimestamps: [00:00:00] Cold Open [00:00:28] Swyx Intro [00:01:59] Call Start [00:03:07] Gatsby v4 [00:06:23] Incremental Builds [00:07:16] Cache Invalidation [00:09:03] Gatsby DSG vs Netlify DPR [00:09:35] Abandoning Redux for LMDB [00:11:50] Parallel Queries (PQR) [00:13:32] Gatsby DSG [00:15:24] Netlify DPR vs Gatsby DSG [00:19:19] The End of Jamstack [00:22:12] Tradeoffs and Performance [00:24:34] Image Processing [00:27:25] Automatic DSG [00:29:33] Gatsby Cloud vs Netlify [00:33:34] Gatsby vs Next.js [00:35:41] Gatsby and the Content Mesh [00:37:19] React 18 and Gatsby [00:39:45] Custom rendering page fragments with React 18 [00:42:10] Server Components in Limbo [00:43:33] Smart Servers vs Smart Clients [00:45:21] Apollo and Open Source Startup Strategy [00:47:06] TMA: Too Many Acronyms [00:49:16] Gatsby for Docs Transcript [00:00:00] Cold Open  [00:00:00] Abhi Aiyer: And so with LMDB in place, right? We have workers that can read and write to LMDB, which allows us to run parallel queries. So PQR was a huge advancement for us. I think we saw up to like 40% reduction in query running time. And build times went down. We had a goal, I think it was like, we'd try to look for at least 20% reduction in build times and I think we hit 26%, so all cool wins, you know? [00:00:28] Swyx Intro  [00:00:28] swyx: The following is my conversation with Abhi Aiyer, and Ward Peeters, two lead engineers behind Gatsby Cloud, and the recently announced Gatsby V4, which is at the forefront of what I think is the most significant change in the JAMstack landscape in the past two years. We discussed how parallel query writing PQR and deferred static generation DSG are achieving 40% faster queries and 300% faster overall builds. [00:00:53] And they did a wonderful job handling the most impolite questions I could think of, including whether it Gatsby Cloud is a Netlify clone or the Gatsby should just be a data layer on top of Next.js and how they're dealing with TMA too many acronyms in web development. This conversation should be viewed together with my past discussions, with Sunil Pai and Misko Hevery in considering the cutting-edge of web development today. Online discussions often present a binary split in that your technical choices either have to optimize for developer experience or user experience. [00:01:25] But I find that it is builders like Abhi and Ward and Misko and Sunil who are constantly trying to improve the experience of developers in building great user experiences by default. I hope you enjoy these long form conversations I'm trying to produce with amazing developers. I still don't have a name for it. [00:01:41] And I still don't know what the plan is. I just know that I really enjoy it. And the feedback from you guys have been really great. So if you like this, share with a friend, if you have other requests for guests, tag them on social media, I basically like to make this a space where passionate builders and doers can talk about their craft and where things are going. [00:01:58] So here's the interview.  [00:01:59] Call Start  [00:01:59] Abhi Aiyer: I'm Abhi Aiyer. I'm a principal engineer at Gatsby. Thanks for having us.  [00:02:05] Ward Peeters: My name is Ward Peeters. I'm a staff software engineer at Gatsby and I'm from Belgium. And I've been working mostly on the open source side.  [00:02:15] Abhi Aiyer: I forgot to say where I'm from. I'm from Los Angeles, you know, Hollywood,  [00:02:21] swyx: I'm actually heading down to LA,  [00:02:22] Abhi Aiyer: in a couple of weeks, there's,  [00:02:24] swyx: I'm going to Kubecon, which is like a very interesting thing for a front end engineer to end up at. But that's where my career has taken me.  [00:02:34] So this conversation started because I had a chat with Sunil, on this podcast that I accidentally launched. I don't think we did Gatsby much, a good favor. [00:02:45] Like we both saw the new updates and I didn't get to say the nice things that I thought about Gatsby. I should also say that I used to have my blog on Gatsby and I no longer do. I used to work at Netlify and I no longer do. There's a lot of history here for me with Gatsby. It's been a while since I caught up, and I'm curious to see or get the latest. [00:03:07] Gatsby v4  [00:03:07] swyx: Maybe we should start off with like a quick summary of what's new with Gatsby with Gatsby V4, right?  [00:03:13] Abhi Aiyer: Is that a good place to start? Yeah, I think so. [00:03:17] swyx: So first of all, I think the marketing was really nice. Gatsby camp, it seems like a really big push and qualitatively very different from Gatsby 3. Tell me about what the behind the scenes was like. [00:03:30] Abhi Aiyer: Yeah, it was, we're getting better at the marketing side of what we're doing these days and Gatsby 4 was a big push. It really changed how we approach the framework as a whole.  [00:03:43] For those who don't know, traditionally Gatsby was a static site generator, purely static. We hold ourselves high on our connections to a content management system. [00:03:55] And we provide a really good data layer there, that takes all those requests that you would normally make to a content manager system, turns them into a, like a store of data that you can then use and query from graph QL. And the big thing that we were hitting before gas before was. Company was growing. [00:04:17] And as more customers were using Gatsby cloud, we started realizing that we couldn't scale to really large sites and large sites is like a misnomer. Like you could be, you could be a 50,000 page site and be considered large given the data that you may have. But we're talking like hundreds of thousands of pages. [00:04:38] And the thing that we kind of realized is not all pages are created equal on your site. Especially the ones from like 20, 15, 20 14, where, you know, no one's looking at that people, those pieces of content, if you're a site with a huge archive o...

Lion King Medley: https://www.youtube.com/watch?v=iH66hZIPVjkFrom Now On: https://www.youtube.com/watch?v=ruCl8X-xWPAhttps://www.musicalityvocal.com/If you like the Greatest Showman also check out their cover of This Is Me

2015: https://www.youtube.com/watch?v=HmrDjvv_ENQ2018: https://www.youtube.com/watch?v=XcqloQezOUgGitLab IPOed for $15b valuation today.

Listen to a16z live: https://a16z-live.simplecast.com/episodes/one-on-one-with-a-and-z-11-how-much-rare-knowledge-is-there-4aF0UbP7- rarely wrong just early- pets.com, diapers.com- hadoop -> databricks- Nycira -> central control plane - 2b revenue company- rare knowledge - what do you believe that nobody else believes- 2 kinds - super secret, or in plain sight- airbnb - history of hotels- new ideas nobody else has- take seriously things that nobody else believes- history gets rewritten to be both obvious and inevitable- bill gates the road aheadTranscript [00:00:00] Hey everyone today, I'm sharing a clip from the Andreessen Horowitz podcast from both Marc Andreessen and Ben Horowitz. Which is them talking about rare knowledge basically the secrets that you believe that no one else does as well as how to develop a view of the future which is important obviously for vcs but i think generally important as well [00:00:19] Marc Andreesen: How much rare knowledge is there in the world in your experience or concretely, how often does it happen that there are less than 10 people you can think of? No. I could do something and you'd be skeptical you can find anybody else that can, and then a huge, no asks. How can I develop a view for the future, which I think is actually a very related question.  [00:00:38] Ben Horowitz: So way more than you would think. There's a lot of very knowledge and, mark and I like experienced this on our job. Every single day. There's just, the world is really dynamic. Now, in fact, it's probably never been more dynamic, I think, by any measure. And so what we see all the time is the old conventional wisdom just ceases to be true. [00:01:02] And we've seen this here. One of our favorite examples is just. All the dot bombs that everybody made, hysterical fun of during the early two thousands, all the idiotic ridiculous, stupid ideas that people had for the future, which were just so obviously. All eventually worked. And and it was a matter of the underpinnings of the internet and other things changing to the point where those really bad ideas, Pets.com or diapers.com or any of these kinds of things, they all work fantastically later, as world changed. And that all was super rare knowledge because the conventional knowledge was of course, those things are all the stupidest things ever, and you'd have to be some kind of moron to leave your high-paying consulting job to do that. [00:01:54] But we continually see this and in the firm, we even have kind of a rule. Which is, if you know too much about something you got to back off, because you know what, particularly, if historically what did not work that can be dangerous knowledge in our business because you can miss it the next time when it actually does work. [00:02:14] And w we just hit all the time on. Favorite investments that I've made was it was common knowledge in Silicon valley that Hadoop had one big data, like architecturally, like that was the thing. It had one open source that had hearts and minds, blah, blah, blah. It was going to be the. [00:02:30] And I think, probably the best, for sure one of the best investments I ever made was that wasn't true. And so it's just like a small piece of rare investing knowledge, but a big piece of rare knowledge for the entrepreneurs who invented spark. And then, later turned that into Databricks. [00:02:46] Marc Andreesen: Yeah. Ben, do you remember? So for we invested in a company called Nycira early on in the life of the firm, like nine, 2010. And then do you remember the we shouldn't name him, but we had a meeting with a a I believe the CTO of one of the really big networking companies at the time in our diligence literally said it was  [00:03:01] Ben Horowitz: against the laws of physics. [00:03:02] It wasn't possible. They had already like steady. At length this very large, important networking company and you could not have a central control plane and the way that the Sierra proposed to do. And of course, now in the Sierra inside, VMware is like a $2 billion a year revenue. [00:03:21] Marc Andreesen: Yup. Yup. And then of course, a classic essence on the consumer side of course, is that everybody knew right up until 2004, that consumers would never put their real identities online. That was the one thing that would never happen. Never. And then of course Facebook like completely blew that open. [00:03:32] So the twist that I want, there's a couple of twists that I wanted to put on. This are circuit aspects of this very important question that I want it to get a little deeper into. For those of you who like, think about these things you will have, some of you will have at least read Peter Teal's kind of famous book zero to one, he talks a lot in that book about, what he calls the secret, which is this idea of the were knowledge that other people don't use. And then he has this famous question that he asks he had kind of dinner parties where he asked this question of what, what is the thing that, that nobody else knows. Or he asks a, there's a sort of a related verse and the question is just, what do you believe that nobody else believes? [00:04:03] And of course what's interesting is those are not necessarily the same thing. And then on top of that there's this question of okay, it's the rare knowledge, something that is actually not known. Like it's actually a piece of information that's like invisible. And so for example, rare knowledge, let's take a hypothetical case, rare knowledge of a chemical. [00:04:20] That has been invented in a lab and only the people who worked in that lab know that formula exists and only they know what the formula is, right. Or maybe the formula for KFC seasoning might be something that's, a formula it's literally locked  [00:04:34] Ben Horowitz: 11 herbs and spices, but we do not know what they are. [00:04:38] Marc Andreesen: We do not know which one. So there's literally the rare knowledge is any knowledge, literally like you can't get to it. There's this other kind of rare knowledge which goes to the second version of Peter's question of what do you believe that nobody else believes her fear that people believe, which is like the rare knowledge of something that is actually like a fact that is actually completely like, basically in, in public view. [00:04:59] It's like a thing that anybody can walk up to or learn about or read about, or read on the internet and tout, or read an academic paper about or whatever. But it's just people just simply don't believe it. Like they just don't buy it. Like they're not having it. [00:05:11] They just, they, they found some reason to rule it out, by the way, they're often a ridiculing it. And so Ben, the question I always think about on this is. How many of the secrets, right? Or how much of the rare knowledge in the world, how much of it is literally the 11 herbs and spices that you can't go find out versus how much of it is the thing that's there in plain sight that everybody's just making fun of? [00:05:30] Yeah. The  [00:05:30] Ben Horowitz: plane tech definitely seems much bigger. At least in, in artwork we see many more of the plain sight ones. One...

Listen to the Ship It! podcast: https://www.listennotes.com/podcasts/ship-it-devops/its-crazy-and-impossible-Q3tD0SVWCh1/ (45 mins in)Steve Jobs anecdote - $50m fumble https://twitter.com/apartovi/status/1447251334814523392?s=20

Video: https://youtu.be/Cxaf8E00GMMSlides: https://docs.google.com/presentation/d/1sJSqNy-t-kVxzrWlqMTp_03nI7Zo8Znr7k0f0C6L9ig/edit?usp=sharingTimestamps:[00:00:00] Intro[00:02:17] Part 1 - Components: Code Organization for Real Apps [00:04:26] What we learned from React [00:07:46] Part 2 - Architecture: Choreography vs Orchestration [00:13:05] Retries and Timeouts [00:14:37] Part 3 - Time: React vs Temporal[00:16:34] Elevator Pitch [00:17:13] Programming Model [00:18:44] Comparing React and Temporal Principles [00:19:11] Live Demo: Amazon One Click Button [00:23:49] Talk Recap [00:24:16] React and Temporal Full Comparison [00:24:42] Conclusion: EnablementTranscript [00:00:00] Once again, I want to thank you all for tuning in and joining, React New York 2021 without further ado, I'll pass it on to Shawn. All right, so hi everyone. Hello, React new York. It is my home town in the U S and I miss everyone back in New York. I am currently based in Seattle, but I'm here to talk about React for the Backend. In 2020 I actually thought that I had given my last React talk because I was all tapped out. I had said everything I wanted to say, and then React New York came by and said, do you want to speak? And I was like, oh, I really wanted to speak for React New York. So here's my presentation about what I've been working on and what I think the parallels have been for React. And I think there's some generalizable lessons, even if you don't end up using Temporal. So, the inspiration for this talk came from Guillermo Rauch, the creator of Next.js. And he was the first person to point out that Temporal.io, does to backend and infra what React did to frontend. Temporal engine is quite complex, much like React, but the surface exposed to developers a beautiful render function and I'm a bit upset because he realized there's before me and I have been working on Temporal for a few months now. So important caveats before I start this talk. What I'm presenting to you is alpha for TypeScript. Temporal is typically a goal or Java based application, but we're developing TypeScript and hopefully launching it soon. And then finally "React for the backend" is an analogy, not a design goal.  The way I treat this is like, it's a, it's basically like crabs. And one of the most entertaining facts that I've ever found is that nature has apparently tried to evolve crabs five independent times. And in fact, there's a word in evolutionary biology for it called Carcinization. And of course, this is really good for a lot of memes. So tired convergent evolution is not uncommon, especially when species have similar selecting pressures in their environments, wired. Everything is Crab. And perhaps everything is React, because we have similar design space problems. So I'll tell a little bit of the story through three parts there's Components, and we'll tell it through the story of Uber, talk about architecture, we'll talk through the story of YouTube, and Time will tell you through the story of Amazon. So a lot to cover, I'm going to try to go really fast. Don't worry. I'll share the slides on my Twitter later on. Okay.  [00:02:17] Part 1 - Components: Code Organization for Real Apps  So part one is about components. You see this a lot on YouTube. Probably you're watching now on YouTube or live streaming. And yeah, you know, like three hour live stream and that's it. Very cool. I think we, we know how to break things down and React has really helped us be more productive by being able to break things down into the components and knowing how to compose them together in a predictable way. But there's a lot of things unanswered in things like this in, in full stack, clones of major well-known apps, which is the hard parts. What like a typical Uber trip, we'll have all these steps like search pricing match. Pick-up drop-off rating tipping, payment, email, uh, and so on and so forth. And typically the naive way of organizing all this is basically one after the other, right? Like this is search goes to pricing, goes to matching, goes to pick upgoes to dropoff goes to rating goes to tipping goes to payment, goes to email, imagine that these are all managed by separate teams and scaled independently. Then you realize, like, this is only the happy path. Then you have to throw in a whole bunch of things that can happen along the way. An Uber trip is basically a long running process with humans in the loop and humans are very, very messy by nature. So how would you write an Uber clone? good luck with a lot of data technologies that you would typically reach for just naively, because you will have to discover all these systems and all these use cases and edge cases along the way. So when people say full stack, they often really mean like this half drawn horse meme. I think this is particularly funny so I take every opportunity I can get to show it, but to be honest, a lot of us front end developers are probably the other way the half-drawn Dragon where we're frontend a very good and in the backend, we'll just like, you know, stick some stuff on Firebase and something. And in reality, if you look at the backend systems, most companies, especially at scale, go towards some form of very complex micro service, system. I don't have the chart for Uber, but Hail-0 is probably a good comparison. Netflix, Twitter, and It's not really avoidable. If you want to scale a company to any significant size, you probably have to break them up into independent services because you're going to ship your org chart anyway. [00:04:26] What we learned from React  The thing I realized as a React developer, as a front end developer, is that actually we had a pretty good run in the past seven, eight years of React in terms of the fact that front end developers know how to organize code at least in terms of the component level. So we moved from the jQuery era where everything was just kind of spaghetti all over the place to at least something more organized where event handlers are strongly tied, locally tied with renders, but essentially managed by React's runtime.  So a few key lessons from React that I personally draw [00:05:00] is that you want to have a component and a renderer model. Like, so essentially the user or the developer writes components. And then the react core team writes the render and that handles a lot of the boilerplate that you might typically forget. And this is everything to do with on mounting or having local states. And it gives you a very nice, non-leaky abstraction that you can write. Second, you can also guarantee work and correctness, which is originally what drew Jordan walk to make something like React because he was working on Facebook messenger and there was a lot of inconsistent state within Facebook manager because of the spaghetti code. So correctness, meaning that we embrace functional programming to produce a virtual DOM view is a pure function of state. If you look at the old enough React talks, you will see a lot of v = f(d), so view as a pure function of data. And finally the programming model. We like to say that it's just JavaScript. There's no custom syntax ...

Squid Game OST: https://www.youtube.com/watch?v=ADyazK7btX4Trumpet Concerto: https://www.youtube.com/watch?v=oYA2RRJpS6kBlue Danube: https://www.youtube.com/watch?v=KEGdn0OZ22Qhttps://www.thefocus.news/tv/squid-game-what-is-the-classical-music-played-before-the-game/

Listen to the Infinite Loops podcast: https://www.infiniteloopspodcast.com/alex-danco-everyones-job-is-world-building-ep53/ (40mins in)Read the full essay: https://alexdanco.com/2021/01/22/the-michael-scott-theory-of-social-class/See tweet reactions: https://twitter.com/swyx/status/1424953927426928646Transcript  [00:00:00] Hey everyone today, I'm about to break one of the core rules of this mixed tape, which is that episode should come in at around 10 minutes. Uh, this is a 25 minute rant from Alex Danco, which is still one of the best essays I've read of the year. He posted it on January 22nd, and I think it's still resonating.  It basically is a theory of social class that is both entertaining and actually strangely true. Just don't think too hard about it because if you do you will realize how true it is. Maybe this is a good segue into the other thing that I feel like we were theoretically supposed to talk about this podcast episode,  which was the Michael Scott theory of social class, which again is like another example of the power of world-building in this  case, it's the power of  world-building as applied to the phenomenon of middleman. Right, right. Like, what is middle management? If not world built real, because that's all you've got right. Is this  world you've constructed. So originally a missile, who's  going to fill the forms out. I mean, come on.  It's not, it's not anything who's going to fill the firms out. It's like, who's going to create the meaning. It's like I forgot who described middle  management as the control rods and a nuclear reactor. It's like, but the point of them is to slow things down so that it doesn't run out of your control and blow up. So,  okay. So I wrote this piece, the Michael Scott theory of social class,  which is basically a re skinning of Venkatesh Rouse article that your base principle, which itself was a re-skinning of Holly White's book, the organization, man. Oh, so that's the actual source material is the organization, man. Have you read the organization, man? Do you know that book?  I don't have notes on it. So that means I didn't take it seriously. It's so good. It's so  good. It's in the list of  books that I recommend to everybody. So I'm going to have, I'm going to have to read it again. What's remarkable about the organization, man is  simultaneously how in a literal sense. It did not get the future. Right.  But at a second  order sense, it just nailed the future so hard. They got it. So, so, so right, just at a slightly different abstraction layer than people realize. So the general thesis of the organization, man,  is that all organizations  that survive have stratified into three layers. You have the bottom layer, the middle layer and the top layer. The bottom layer is the people who do the actual work. This is the majority of people. Their lives are spent doing literal things. So these are line workers, frontline people,  anybody who is  actually producing something. Literally there are the people at the bottom there, the majority of  people at the top, you have the exact. They actually have a lot in common  with the people at the bottom  in the sense that they have very  literal roles and responsibilities and very real stakes involved. And they see the world very clearly as it is, but the people at the bottom and the people at the top see the world through clear eyes with clear actions and consequences, but there's this group of people in the middle called middle-management that is really, really different than either of those groups. And their job is to intermediate between  the people at the top and the people at the bottom by basically constructing this reality called middle management that does not literally produce anything nor have any literal  stakes or consequences, but whose job is  effectively to mediate like the control rods in the reactor to say like, look,  the goal here is to create a stable system  that perpetuate. Regardless of how efficient it is or how complicated it is or anything just like, can you get something to persist? This group of people will always emerge in one form or another. So in the 1950, in the early  fifties, when Holly white wrote this book, this was in the era  of these mega mega conglomerates, like Dow DuPont, us steel, general motors, like this was the field. Like the current mindset was that. The frontier of progress was mega organizational dynamics. It was scaled to get scaled, to get scaled. This is how everything works. Eventually everything will be run by four corporations because we figured out the science of how management works. And specifically we figured out what middle-management. We created this whole world  of middle management that has sense of purpose and a sense  of identity. And it was fed through these institutions called business schools and the NBA. And this whole  idea that like middle manager was actually this craft more or less independent of the  industry. It's like, what do you do? Oh, you're a manager. Oh, like what kind of industry do you manage? It doesn't matter. Like I do manage. Right. That was the thing that you can learn. Do you go to management school and learn management regardless of where you were from? You remember in the office?  I'm just sitting here because I love that movie. There was one particular episode of the office  where David Wallace, a CFO brings in a new boss for Michael. Who is Idris Elba who comes in as like the professional manager. And Michael's like, where are you from? He's like, oh, steel.[00:05:00]  Um, but anyway, so this is idea. It's like, like as Holly white describes  it, William H. White, he went by Holly white as  he describes it. Right? So the whole book is about this three layer system. And this three  layer construct where you have the people at the bar. Who again are doing all the actual work and have no path to leverage. And the people at the top  who have all the leverage, but  are very deeply suspicious of everything and are perpetually trying to acquire and keep control over this huge sprawling.  Basically those two sides in order  to not fly apart at a hundred miles an hour, need this mediating influence in the middle called the middle management. <...

Listen to Brett Goldstein's podcast https://www.listennotes.com/podcasts/films-to-be-buried/bill-lawrence-films-to-be-vyfQWE4F5O3/ (20 mins in)