Beyond the Network: Unlocking the Power of Programmable Traffic with Ngrok

Richard Moot: Welcome to the Square Developer Podcast. I'm your host, Richard, head of developer relations here at Square, and today I'm joined by Peter from Ngrok. Peter, thank you so much for being with us here today. I'd love for you to just give us a little bit about yourself, a little bit about Ngrok, and let's kick things off.

Peter Shafton: Sure, hey, thanks Richard. Thanks for having me. So my name's Peter Shafton. I'm the CTO of Ngrok. I've been with the company for a little over three years, and I actually learned about Ngrok way back from when I was at Twilio about 13 years ago because the founder of Ngrok, Alan Shreve, was an engineer on a team there that I was running at Twilio. So I first started the voice and messaging parts of Twilio, that was actually the beginnings of Ngrok. But I've been sort of bouncing around Silicon Valley for a lot longer than that. A bunch of companies you all have heard of, probably Cisco and Yahoo, those who are old enough, SGI, and then a bunch of little startups that people maybe didn't hear of as much. But yeah, that is my past. So I'm a developer at heart, an engineer at heart, and then somehow ended up doing architecture and running technology.

Richard Moot: I feel like they always trick us and lure us into this in some way. I mean, it's very alluring, but then eventually you're just like, wait, what happened? I'm now the CTO of a company.

Peter Shafton: That's right. As long as they don't take away the keyboard, I can still type code there. I'm happy.

Richard Moot: Do you still get to occasionally write some code for things or do you find the time for it?

Peter Shafton: I do. I do. I do it badly, and most of my engineers hopefully don't let me do that. I end up in the data space more than anything or through SQL or the data warehouse and data lake areas, but at times I'll write some low level code and networking code and things like that.

Richard Moot: I mean, you got to find the time for it. It is kind of invigorating, but at the same time, I would agree that outside of Dev Rel, I am less inclined to build stuff that's going to be relied on in production, and I'm more inclined on building, here's how to do this particular thing, or here's a little script for pulling some data together so that we can make sense of stuff.

Peter Shafton: Yeah, I think most of us got into this because we loved writing code. I think if you'd asked me, I was a young kid, an Apple++, and I didn't realize this was a career. I just thought it was a really cool hobby to have, and then the thought that somebody would pay me to do it, and so this is the piece that got us excited. If you left most of us alone, we would still be writing code. It's just nicer when you're doing it at scale and allowing other people to see what you built versus,

Richard Moot: Yeah, no, I mean I think that it's really well put. I mean, most of the time I'd build tiny little automations in my house for like, oh, now it's much easier that when I want to go to bed, it turns all my lights off at one time instead of just me having to go through all of them. But now it's like now I can solve problems for thousands of people, millions people.

Peter Shafton: Oh yeah.

Richard Moot: So you used Ngrok a long time ago before you actually even joined the company. I'm curious, what was it when you first used, well, two things I kind of want to answer. For anyone who's maybe not familiar, we can give a quick little explanation of what Ngrok is and then what was it like when you first used Ngrok that made you fall in love with it, or I don't know if you'd call it love, but I mean I felt like I fell in love with it when I first used it.

Peter Shafton: No, it's a good story. It's a good story. Yeah. So there was an engineer by the name of Jeff Program. He was one of the early engineers at Twilio at the time. He had come from NASA and he built a tool called Local Tunnel, and he built it for a very specific purpose. So Ngrok basically fit into the webhook infrastructure that was Twilio. So the early days of Twilio, it's still true today is all powered by webhooks, which effectively means you get an inbound phone call, you get inbound text messages, you want to respond to that and control the programmability. You had to respond with XML, with twiML at the time to an inbound web request because that looked very much like how the internet worked. They figured people were writing PHP scripts or Python code for a website. They were used to getting a form post and responding, they thought, you can just reuse that infrastructure. It just happens. It's a telephone that's calling you instead of a browser client making the request, which made a lot of sense because that was what the developers were doing and it was easy to build on.

The challenge was if you're inside Twilio and you're building the product and you need to be able to test it or even iterate on what you're doing, a very slow loop is I build a thing, I deploy it to production, I let a webhook hit it and see if it works or not. What we wanted to be able to do was change the code, maybe even sit in a debugger and have the webhook come in and trigger our code. And so to solve that problem, Jeff built something called Local Tunnel, and it was a simple bit of Ruby code that effectively gave you a dynamic URL.

So if you think about it, they call it reverse proxy, but you basically have a little sidecar sitting on your machine. It makes a network connection out to a server that is on the public internet and it tells that server a domain name to use, so fubar or whatever it is. If you hit that request, it'll basically tunnel the traffic back to your local machine so you can iterate and have something that kind of looks like it's on the internet, it's accessible on the internet, but it is not otherwise there. It's on your local machine and if you shut it off. So that was local tunnel and it was used inside Twilio. Alan was one of the engineers at Twilio using this, and it was fine when it was just a few of us. As it scaled up, we realized the Ruby implementation was not particularly scalable, didn't really serve our purpose at scale. And so he said, I think I can rewrite this and I think I could do it much better. And he started to iterate on it and go, and when he left Twilio, he said, I'm just going to make this a side project. It was, but it turned out it was a side project that went very, very far. So the early implementation of Ngrok was very much a tool for testing, for testing webhooks, for developing websites, developing APIs, and that was kind of the early history of the product and of the company.

Richard Moot: And that's exactly where I probably first started interacting with it when it became this sort of tool that was put out there. And I remember even really early on, it was very, I mean, it was straightforward for signing up if you actually wanted to use it beyond the sort of free version of it. But I got to say that after just installing that CLI tool, putting it in your local bin folder and then just calling it from your path wherever you want. And I think the first time that really got me was when I was working on something at a company and I was trying to update stuff on a marketing page and trying to get whatever it is that they wanted, the way that they wanted it. And I just ran Ngrok and had a URL, and then I was able to send it to them and say like, Hey, is this what you are wanting it to look like? And they were just immediately giving me feedback and testing things and I could see all the traffic of what they're doing in there. And that was a really amazing thing because normally I would've had to take all of that, push it somewhere, put it up on a staging version, deploy it somewhere, and I really was not at that point, I was like, I'm just trying to move things from one place to another and I want to show you what it looks like on my machine. The ease of using it was just so great.

Peter Shafton: Yeah, no, it made a lot of sense. I think in the early days it still does for different reasons, but the early days, sort of the aha moment, we have a tagline now we use that says online in one line, which basically meant you could download the Ngrok executable type in a simple command line like you might for Curl or something and you'd be up and running. And I think the fun part was both testing for folks like you and I iterating on code turned out, and Richard and I were talking a little bit before this about having automated home infrastructure and those of you who have put up stuff in your house and automated it and realized, well, if I want this to be accessible when I'm not at home, then I need to somehow punch a hole into my home network.

And that often meant messing with the settings on an Xfinity router or AT&T or something else to figure out how do I open this thing up and allow access. And Ngrok made that really simple too. And so we found a lot of developers that were hobbyists and that were doing and automating things. And that sort of device gateway or access became the early days of how it was used.

I think the scary part is once you put something on the internet, you're like, okay, this is out there. Then you forget that it's on the internet and everybody can hit it and use it. And so having other sets of capabilities that do allow you to put authentication in front of it, all of these became the obvious kind of next steps. And if you look at the history of the company, you can see where that progress happened. But it was always that sort of aha moment that I think got early developers and I mean, it's crazy sort of the scale of this hit. There are over 10 million developers that are using Ngrok and thousands every day that come down, which you think of somebody that was bootstrapped, it was hard to imagine that you would start that way.

Richard Moot: Yeah, no, I mean the growth in the evolution of the product has really impressed me over the last few years because initially it just seemed like, oh, it seems like such a simple straightforward tool. And when you're a developer just trying to say like, oh, I'm just trying to build and test my webhooks and I have to expose those to the platform that's going to call them if I want to do any kind of local testing. And then you just go, okay, that's it. But then I think there's other people who just went like, well, what else can I use this for? And you really start, tell us a little bit more about how those use cases started to really evolve. Because I think it's very interesting to sort of understand, yeah, I'm punching a hole. I can do Webhook testing or I can let somebody see my local development, but this ability to communicate within the same network in a secure way is pretty common for a lot of really large apps.

Peter Shafton: There's a lot of things that are really interesting, and it paralleled a lot of the story of the time I had at Twilio as well, which was if you build a platform, it was sort of built of these basic primitives, but it was programmatically driven like you had an API to manipulate it or set it up and you could scale it from anywhere. It enabled all these use cases you're alluding to. And so that is sort of the fun of what Ngrok has become and continues to become in the simple sense you're testing, but if you think about it, at some point you have to handle webhooks in production. Often you're connected to Square, you're going to handle webhooks from that API, they have to go into your infrastructure. You want to manage them and why stand up another infrastructure to do that?

The benefit of being able to shut things down or keep them up or scale them. A lot of the things that we've built and continue to build echo the problems I had at Twilio, if you think about when you first put an API on the internet, you're like, well, my first problem is just making it accessible. You're like, great, I've got it up there. People can hit it. This is wonderful. And then there's sort of that uh-oh moment where you say, well, wait a second, a lot of people are hitting it and I have to probably rate limit them, right? I don't have capacity to handle all these people and I don't want to throttle my request to one customer because another customer doesn't have enough bandwidth. And so I need rate limiting. I have to rate limit by customer ID, and maybe I have some customers that pay me more and I want to give them more capacity.

And at Twilio, I had to build an API gateway team. I had to build a team that could do that. And then the other problems you start to have is, well, gosh, I have customers that aren't just in my city. They're actually hitting me from all over the world. And the latency some of them are getting is quite bad because they're coming in from Asia or from Europe or from somewhere else that don't have infrastructure there. And so at Ngrok, we literally had infrastructure to support that so that you could go from just running it locally to running it remotely. But I think the fun part about having a platform like that is, and I think you alluded to this, is people just auto discover it. I'll give you an example of a story. There was, and I won't tell the name of the company just to make it more fun, there was an IT guy who works at a company that sold a food product and expected to receive mobile ordering, but they didn't have a mobile ordering infrastructure yet.

What they had was a bunch of brick and mortar stores people would come into and order things like coffee. And he was tasked with trying to figure out a way to make this work, and he grabbed the Ngrok Executal and stuck it on his point of sale system that was running Windows. And lo and behold, not only did that give him an API web endpoint that he could hit, but it gave access to his database and he could look at the end of day sales for the store location by pulling down the database at night and he could push updates for discounts and product in the morning. And lo and behold, he could drive all them over. And he sent an email to Alan. He said, I just did this thing. I downloaded Ngrok, I can do this. And Alan said, sure, it wasn't what I thought about when I first built it, but it works for that.

And lo and behold, that store grew to 28,000 locations and that company does millions of dollars a day of mobile ordering across Ngrok because those store locations, if you think about it, have home networks like you and I do. These are not, they often use some cases, a cellular connection. In some cases they were using AT&T or Comcast or whomever. And so he needed to be able to have a reliable network connectivity. Then he didn't want to go and update the routers on each of those locations or convince the IT team. So it was sort of a fun, interesting use case. I've seen people control smart mirrors, remote control boats and planes that they remote into and give access. And these things just have cell cards in them. They're going over the cellular network. So it's sort of fun to see both the things people have done that are hobby and interesting and unique, but also the places where people have used this and built entire businesses on top of it because connectivity was key for them.

Richard Moot: Yeah, I mean, it is one of those things where it's like, I think it's like when thinking about the layers of an app, and I think it's often that the network layer is one that, I mean, I know when you're dealing with really large scale apps, you inevitably always think about the network layer. But I think when we're first building out a web application, we kind of just take the network layer for granted of just like, yeah, it's going to be exposed in some way, and then we have an auth model. But there's so much more that you can do there that, I mean, I'm just imagining that with Ngrok, that it's so much more possible to say, yeah, we have a private network that most of our application infrastructure is communicating on. And occasionally it would be great if for something that's say existing in a more hostile environment, say a storefront with their own networks that are connecting to devices, we can only trust so much.

And it's like, but you have this one machine, the point of sale where you're like, I want to be able to punch a hole for just this specific device and it can talk to my private network through this particular endpoint that I've created here for it to communicate. And that is actually, not a lot of people have to deal with this, but I feel like it's one of those things, it suddenly goes, oh, this makes this so much easier than having to think about mobile device management and certificates and certificate pinning and all these other security mechanisms that you have to employ when you're dealing with all of these different devices.

Peter Shafton: You're exactly right. And it's interesting because we really saw, what we have seen is about four or five use cases or traditional, the device gateway, that sort of connectivity to remote devices. An interesting one, an API gateway has been another interesting one. I'll give you an example. When I was at Twilio, we acquired two companies that a lot of people had heard of. One was called SendGrid that sent email, and another was Segment that did data processing. And imagine you acquire companies that are fairly significant. They each have significant data infrastructures. They already had a footprint in GCP or AWS or in SendGrid's case on-prem, like a private data centers in Colorado. And when each of those companies set up their infrastructure, they created private IP ranges. And when Twilio acquires them, gosh, we should have a single API, right? You should be able to send email to SendGrid via the Twilio API. And you think about that, well, that's hitting the Twilio endpoint. And you might say, well, okay, all we'll have to do is things that are slash twilio slash email should go here. How do I do that? They have their own ingress points, they have their own, and they're in a different environment. I can't just take the DNS that goes to one place and send it somewhere else. I can't split it in the middle and say things that are -V1 should go here. All the complexity of the network and the great intelligence we had ahead, we had totally overlapping private IP space.

So I couldn't just route for one infrastructure to the other because the IPs were redundant that it was not clear where I was going with the traffic. And so Ngrok, if we had had it at that time, would've enabled me to basically say, Hey, listen, I want to shard this anywhere I want. The network is effectively as programmable and configurable as the other parts of my software stack are. And I could have said, Hey, my V two version of IAPI, which will be somewhere in this path, is going to route to my new infrastructure that guess what I've stood up in the Azure Cloud or things that match these headers or these parameters, I want to send this other way. And so being able to do that through a programmable network interface was really what is powerful and interesting about Ngrok. And there is an infrastructure today that we built that basically uses traffic policy.

And the idea is that all of the information that's available to you on an inbound request can be used in rate limiting, in routing decisions, in decisions as whether to force somebody through authentication flows. And that allows customers of ours to solve problems we hadn't even considered today. So we see customers who are doing exactly that, sending some traffic to different infrastructures to challenge certain users with authentication or add special headers or modify their payloads either inbound or outbound based upon what they see. And one of the fun things we worked on was to scrape the web for public data sets about IP addresses. So you know, have geo-data, everybody's kind of familiar with Maxmining, and other places, well listen, given an IP, I can figure out the country. But it turns out people publish the IP addresses of TOR exit nodes, they publish the IP addresses of block lists or the ranges of IPs that belong to Square or Amazon or Microsoft for their services.

And you can annotate those IPs as they come in and you could say, Hey, listen, not only am I going to validate the square authentication token and signature of their webhooks to know it came from Square, I can also tell, Hey, it came out of a Square IP address. And so it's really hard to spoof this and all of that information. Not only do we use ourselves to protect our platform, but is available to our customers to use in really interesting ways. And so that's been really fun to see. Some people are used to blocking countries, but you can imagine doing things like when a request comes in, I can look up based off the authentication, which customer that is. Are they a VIP customer? Should I send them into another infrastructure? Should I provide to them a different rate limiting rate? And that's been sort of fun to see for us. It's a platform for everybody else. It's a playground.

Richard Moot: Yeah, no, totally. And I mean, I feel like one thing that is interesting about what you've sort of touched on in there that when I first joined Square seven years ago, one of the first things I did was I was immediately starting to build something that's going to be making use of our webhooks. And as my knee jerk reaction was to use Ngrok to expose an endpoint and then register that and then start seeing my feed of events happening in the system. And one of the things that I think most people might not understand at first, because when you first say, Hey, we're going to punch a hole in the network and it's going to be publicly available, you're exposing something onto the internet. I think that the reality is it's more secure than you might think. It's, and I'd love to touch on the ways in which Ngrok is sort of, I don't know if I'm stretching and saying secure by default, but I think what ways is it sort of secure by default? So when people use it, they're not necessarily worried like, oh my gosh, somebody's going to be able to call into this and start seeing stuff on my system.

Peter Shafton: Yeah, there's a bunch of different levers you have. So you think about it, and it's interesting because it kind of depends on people's, you see in the old days or the first early days, you could have an HTP or TCP endpoint, which kind of meant you could basically have an IP domain name and a port that was open to everybody and well, that's very powerful. If you're fronting a thing that you just want to get online, we already know about HTPS and TLS and SSL as a mechanism to know at least you're talking to the thing you think you're talking to, right? And part of that is, is there a certificate for it? Is that certificate up to date and valid? And if you've been running network infrastructures, sometimes your certificates expire and you have to renew them Ngrok takes over all of that part of the management, right?

The second thing you can think about is the traffic transiting the Ngrok network secure in another way, sorry, secured through TLS, but where is TLS terminated, right? Do we terminate on our edge? Does it need to be terminated all the way back at the service that's actually handling it? And at Ngrok, you can choose where you want. So we have customers that do zero trust, that basically we don't see their certificates, we do not terminate the traffic, we route it through and it's up to them to terminate. But in the world where, and there's a bunch of protections you can do with that.

One example is you might know the IP addresses, these requests are coming through and you can whitelist those and say, Hey, listen, these are only IPs I'm going to allow or there's a set I want to deny. And with the IP intelligence, you can effectively say only from this ASN or other pieces that make it even more flexible. There's DDoS protection if you think about it. It's like, okay, put this thing up. Am I going to get hit with a packet flood? And so we have a firewall layer that does detection of SYN packets and it does it globally. So it has to basically know, okay, maybe it's an attack that's actually being spread wide enough across the world that any single node doesn't kind of see it, but in aggregate I do. And so that is a piece that we have both to protect ourselves and to protect our customers.

The other use case, we talked a little bit about authentication, you can turn on authentication and say as an example, Hey, I want to use Google Auth and they have to match this domain as an example. We use Google Auth for Gmail and other pieces. And so if you don't come in with an Ngrok.com domain and authenticate through Google, I don't let you in, but you can imagine you can also use a subset of emails within that domain. Maybe it's just mine, maybe it's just me and you, Richard. And so that's another way. And then when you're doing machine to machine communication, obviously the machines are not going to authenticate through that, but they can do mutual TLS and in that case they have a certificate that they have to mint on the client side and validate that that's coming in. And so all those are the building blocks that kind of allow you to secure the edge that you have.

And then the last piece of it kind of becomes observability. And so if you've used Ngrok in the early days, you kind of remembered you could hit port 40-40 on a local port and you'd have this browser window and the browser window showed you, Hey, this is the traffic. Not only could you see it, but you could actually kind of replay it. So as you alluded to, somebody was testing a webhook, you're like, great, my code is broken. Can I just replay that? And you could, we've moved effectively all of that logic server side. So now if there's a team of developers, you can see all of the traffic that is coming through, you have the IP intelligence on that data too, and you can kill sessions and block sessions. So if you're doing authentication through us, you can say, no, I'm going to take Richard's token away.

And any track that he had will cease to be functioning right now. And I can see the history of what he had done. And all of that was pieces that we thought were really interesting. And then conceptually, you can replay through that same interface. And all of these were built effect to solve the problems Alan and I and others we're seeing in our history over the last 30 years. But the stuff that developers are coming to us and telling us like, Hey, I'm seeing this. I need you to be able to do this. I need you to be able to scale this way. That's really the power of the platform.

Richard Moot: No, I mean it's really impressive. I mean, there was several years ago when we had acquired Weebly, and as part of trying to get them integrated into the rest of our infrastructure is one of the primary ways that they would communicate was through our own APIs. So as you could imagine, one of the things that most of the engineers working on that integration process was having to integrate with our webhooks and the go-to thing was like, yeah, you spin up Ngrok. And then once we were able to actually get it working within our networks, it was a more secure way for them to be able to just spin up, start seeing feeds and building into our systems to sync databases between Weebly and Square online. But yeah, so when Weebly was onboarded, I mean one of the things that we pushed really hard for was having better tools for doing these integration processes.

And it is really great to finally be able to use this more predominantly and also recommend it to people coming onto our platform externally. So third party developers who are like, Hey, I really want to build an app on the Square developer platform, our top recommendation is usually to just use something like Ngrok to be like, if you run this, you can start building against our webhooks. I know previously we've all probably used, what was the name of that site, I want to say it was website.webhooks or webhook.website, or it was basically,

Peter Shafton: WebHooks.fyi

Richard Moot: Yes, where you would just can drop it in and you can see events come through. But the problem that I've always had with that is, yes, I can see the events coming in, but it's not running against my code. My code is not reacting to any of the webhook events. And so I'm not seeing the additional parts of this of actually seeing is my code that relies on the Webhook events working appropriately. With that, I also wanted to just plug one other thing that I'm excited to try out. I will be forthright in that I have not tried it out, but the SDKs that you have for actually using Ngrok in application code, which I will admit it kind of hurts my brain a little bit to think, okay, so the code itself is not going to be aware that it's actually really tunneling out traffic and responding to stuff through here. But tell us a little bit more about these wrappers that you have to enable that kind of integration directly within somebody's code.

Peter Shafton: Yeah, there's two fun pieces here. So the first is you alluded to, most people who are familiar with Ngrok or have used it in the last decade are used to downloading an executable. You're like, Hey, did you download Ngork? And that's this traditional thing. It's a separate executable startup. For those of us developers that might not be a big deal if it turns out you're building with Ngrok and you want it to be part of the infrastructure, you're standing up sometimes you don't want your customer, maybe as you all alluded to, you're building a Square app. You don't want to tell your customers, Hey, go download Ngrok to use this. And so we built SDKs in a bunch of different languages, Rust and Java and JavaScript and Go and Python that you can just build the capability. And what Richard is alluding to is we did it in a way that effectively in most of your code, you're used to standing up like an HTP server or something like that, and those servers are often based on basically a TCP socket, a listening socket. Effectively they open Ngrok for all intents and purposes, looks like a listening socket. So effectively these libraries provide a socket that just can be handed to the HTP server objects in Java and Python and Flask, Restful and all these things that the framers you're used to playing with. And so your code isn't aware and conceptually could choose on the fly to be Ngrok enabled or not. And the interesting part about that is often when you're running locally, you have Ngrok there because otherwise you're not accessibly can't get requests from the outside world. And when you push to production, the incentive is like, well, now I'm re-architect the thing. I'm going to put engine X or some other form of ingress in front of it and talk to the IT team to get it all set up.

The answer is, if you built it this way, you don't have to do that effectively. You can just push the executable up and it is actually internet enabled at that moment because it is an outbound socket that's powering it. And so that's really cool for the SDKs and the use cases that people ship it. There is another part of that story that I think is really fun and interesting is a PCLC called the Ngrok Kubernetes Operator. So in this world, you basically have a helm component that you install, and there effectively is a service within your Kubernetes cluster that acts as the ingress to that pod. And so when people are used to using Kubernetes and using the Gateway API or setting up an ingress or load balancer, they basically leverage Ngrok to do that load balancer. So now you don't have to worry about, is my Kubernetes infrastructure running in AWS and I should use ALBs and NLBs, is it running in Azure or GCP?

Is it running on my local machine? So you can basically have ingress to services and the same definition of endpoints. The other fun part about that mechanism is you can effectively tunnel traffic from your local machine into a Kubernetes cluster. So imagine you have a Kubernetes cluster running in production or in staging and it has a bunch of services running, but one of them you want to iterate on that workflow normally would look like I work on my code, I write the Docker file, I deploy it to that cluster, and then I test it and realize I broke it again. In this world, effectively that service endpoint that's in that cluster just tunnels the traffic through to your local machine. So it's sort of how you were using Ngrok today to say, my service will be visible on the public internet. Now you can say my service would be visible in a Kubernetes cluster, and that allows you to do things what we call private or internal endpoint.

So if you think about, Hey Ngrok, I put this thing up, it's on the public internet, you can actually say, Hey, Ngrok, I put this thing up, but it's actually not publicly accessible. It's only accessible from within this Kubernetes cluster and other services and pods within the cluster can hit it, but people from the internet cannot. And if I want to expose my local machine, then we manage the mutual TLS certificates that guarantee that the traffic coming into hit your machine is actually coming from that cluster. And so those pieces are really interesting building blocks as well because we see a lot of people who, their deployment model actually is Kubernetes, right? They're actually deploying pods, they actually workflow, and this works much more elegantly for that use case. And so that's a piece that's been fun as well, and it works in a similar fashion to how you think those SDKs work. And guess what? It's actually using our Go SDK, right, so it is both a consumer and Ngrok. The fun part is we dogfood everything. So our API, our website, our docs pages are all fronted by Ngrok. So they are really limited and protected and globally redundant by our own platform. It's a little meta if you think about Ngrok's behind Ngrok, but it is.

Richard Moot: But I mean, I find it hard to find a better way to really iterate because on a product, because you're customer zero constantly. And so if it works, it's working for you, it hopefully is working for everyone else and you can continue to improve and expand, but you're still serving customer zero, so you're not breaking yourself otherwise you're going to be in a lot of trouble.

Peter Shafton: We had a funny story. So we put the website, we had a lot of the components behind Ngrok already, the API and the dashboard. We hadn't yet put the website Ngrok.com. And when we did that, four hours later we got hit by a DDoS attack and we were like, okay, this unexpected. At first we thought we had done something wrong in the deployment. Turned out, no, we were getting hit and we looked in our observed, really realized the traffic was coming a lot from block lists, a lot from Russia and China. And so we said, well, okay, we flipped on a bunch of the traffic policy protections to block tour exit nodes, to block things on firehole or other block lists and Russia and China explicitly. Four hours later we got hit by another DDoS attack. And lo and behold, at that point, it had no effect on our infrastructure.

So it was sort of fun to see how quickly we got inside and I would say it was about 7,000 times the amount of normal traffic we see on the website. So it was clear as much as I know Ngrok is popular and I love to see spikes of traffic, that was not something that we expected, but it was a funny use case. Normally that would've taken days to figure out how do we thwart this, how do we spin up enough infrastructure to protect it? But the pieces were there, we just had to turn the ball.

Richard Moot: Yeah, no, I mean that's amazing that you can be able to react so quickly and make those changes on the fly like that and then just immediately see a result of like, oh yeah, we see the next wave coming and suddenly we're not affected anymore.

Peter Shafton: Yeah, no, it was fun. It was cool. I mean, as much as it's not fun getting attacked, that was fun.

Richard Moot: Yeah, I mean I think with time we can realize it's more fun, I'm sure in the moment it's like, no, we got to get this problem solved and we got to get it solved fast. So I think the final thing I want to touch on is sort of switching gears a little bit, as succinctly as I think we can try to do it with the time that we have remaining, is I'm curious how things have changed for you over time as you've gone through the years of as an IC engineer moving more into architecture and now as the CTO, how has that evolution sort of changed for you in terms of how you approach engineering problems? And tell us a little bit about that.

Peter Shafton: Yeah, it's a good call. I mean, I think as an IC, you're sort of in learning mode. Everything you're doing is your first time maybe, and your goal is to absorb from everybody around you. There's no illusion that you yourself are going to build everything. You're just going to learn from the people around you. And I think as I've grown, that's what I've done both learn how do you grow an organization and how do you grow an infrastructure and what kind of things, what is important when in a lifecycle? When I joined Twilio, there were 20 people, and when I left in 2022, there were 8,500. And so there was that arc that was how does the org grow over time? But it is also what decisions do you make as you're building the infrastructure? And I think those are the pieces you learn. How do I build an organization?

What kind of engineers, what skillset sets should they have early on? What do you want to tell them to optimize for? When you're building a product like Twilio or even Ngrok, the reliability at scale is very, very important. I remember engineers coming to me and saying, I want to build this in Ruby, it'll be much faster for me to execute on it. And I had to tell them at Twilio, you can't build in Ruby because this has to scale for hundreds of thousands of phone calls, calls we build in Java or C, Go so that our customers can write in Ruby. And so there was that part of a learning of which, but that's not always true, right. There are some things that's better to just build quickly in a scripting language because you don't yet know if it's a good idea. You don't yet know if it's going to work, and so Ngrok has been interesting. For me, it's a little bit of a chance to step back into technology and into things with the foresight of what will happen five years out and seven years out, what would it look like if our traffic quadrupled or we land a single customer that's bigger than all our customers. And I think those are the learnings that allow you to make some decisions that you know you can revisit. You'll hear people use the word two-way doors, and if it's a thing you can kind of back your way out of in a reasonable amount of time, then don't wait and think about it a bunch. Just do it.

Richard Moot: Yeah, just go.

Peter Shafton: And if it's a thing that like, okay, this is going to be a really expensive decision if I'm wrong later, those are the ones you sit and haggle about because as an engineer, especially building a company, time is the only enemy you want to move quickly.

You want to try things out, you want customers to get, they love that, right? That sense of, and I see this at Ngrok and I saw it at Twilio. Jeff Lawson, who was the CEO, used to joke. He's like, I want to be the train that's a mile ahead and pulling away. And that is very much true of a startup. You want to be building features and capabilities and listening to your customers iterating quickly. And so as a CTO, my job is to keep priorities front and center to the engineers and let them know this is a thing you can do fast and this is a thing you gotta think about, but that part is fun.

Richard Moot: Yeah, I think one thing, and tell me if this sounds true to you, is as time goes on, you start as an IC, you're focused on the engineering problems of what code should I write, what language should I use? And then as you start going further and further in the architecture, it's like, oh, well what services and what should the architecture of this be? But I feel like there's this moment where you start realizing the biggest problem to solve in this equation is the human element. And you have to start figuring out how do I actually structure teams so that they work together better and organize them in ways that they can focus on the most important problems? I feel like sometimes people can get really hung up on like, oh, but if I write it in this way, I can make it work in so many other ways.

And I think you touched on it right there of saying, well, we can't write code and Ruby because as we go through the abstractions, it's going to be too slow when we have a customer down over there writing stuff in Go. And so we need to be in the fast, reliable, low level type thing because somebody else is going to abstract on top of it. And that requires that human element of helping people to understand there's a bigger picture and there's layers to this whole thing. And how do I organize people to make them come together in the right way?

Peter Shafton: I mean, you're very much right. People love to think of software as this technical endeavor that you can use AI and build your way out of that humans will no longer at some point be necessary. And the reality is these systems are still very much human systems. And so you have both the learnings of the people you hire, which experiences did they have that you didn't in technologies or solving related problems, how do they interact with each other? Who is listening, who who's interacting, and what gets them excited? An engineer's going to do a good job if they're excited about the problem they're solving. They like the space they're working in, they feel like they're doing something that they themselves would've used or leveraged. So it's fun to work on developer tools. Your customer is probably your friend or even you, which is ironic. And then the second motivation is do I feel like I have freedom to kind of solve within some confine as an engineer myself, there's an instant ability for me to look at something and say, I know how you should write that, but I know as a leader, I can't do that because that's disempowering to engineers.

And so they need to be able to be given the context of what has maybe a suggestion of where I've seen walls historically myself, but then the freedom for them to think and solve the problem in their own way. And that's what builds really powerful teams as leaders, as ironic, it sounds your job is to make yourself obsolete and unnecessary, and that's where it has to be. Allowing smart people to run and grow allows me to step out of half of those problems, right?

Richard Moot: Yeah, I think what you said resonates. It just resonates so much with me because I had an engineering manager that I worked with previously, and I had discovered this architectural problem that I realized it was like we were seeing reflections of it in different issues that were arising, and I finally was able to go like, no, this is the root cause of why we're having this problem, and it was this inherent design problem. We're just actually designing these models in the wrong way, and I figured it all out. I wrote it all up, and I even had several proposed solutions, and one of the things that he said to me, which is what you just said right now, is that even though I figured out the whole problem, I'm not going to win people over or get anybody wanting to work on it if I tell them the solution that they need to go implement.

I just need to present the problem as well as possible so that a team gets to go and grasp the problem, see things that I didn't see, but the joy as an engineer is coming up with the solution, and that's how you empower and allow people to grow is that you might see like, hey, there's this huge problem over here. I'm not going to solve it, but I am going to figure out what the problem is and put the right people on it, and then they're going to go figure out how to solve it. That was a huge unlock to me of it is not enough to just be right. It's not enough to just find, oh, there's this thing wrong. You have to figure out how am I going to rally people together to believe that we should address this problem and get them to want to solve the problem and show why it's going to be valuable. Those are all the human part of engineering that it's a form of problem solving, but it's not with code.

Peter Shafton: Yeah, this is the hardest part about going through the engineering ranks. You miss the days where you were coding the solution. You have to be okay with that. You have to get your satisfaction from some other place. It's not going to be you writing all the code and solving all the problems yourself. That is not how you grow.

Richard Moot: Unfortunately. Well, and I've actually, I've heard, I don't remember where I came across this, but it was an interesting take on why the problems that you solve actually never get easier. Because as you get better and better at solving these problems, it's only the most difficult problems. As you've grown as a leader, you go further up and you're zooming further out, and the problems that eventually make their way to you always feel like the hardest problem. And it's like, well, for good reason because you've built these teams that are going to solve the easier problems. So only the hardest problems are coming up to you, and those are the ones that you're having to solve.

Peter Shafton:No, you're exactly right. It's funny, but true.

Richard Moot: Yeah. It's one of those things where it kind of hurts because you go, it's never going to get easier. It's actually supposed to just continually be just as hard or harder as it goes on.

Peter Shafton: Yeah, somebody told me as a CEO, you only get presented the hardest problems at the company. You have to be okay with that.

Richard Moot: So I see that we're pretty much up on time here. I want to thank you again, Peter, for coming here teaching us a little bit about Ngrok. I'm super, super excited to go and try out the TypeScript SDK, and my dream is to actually put this into a CLI that somebody can use and immediately spin up test their webhooks with Square. If people are interested in learning more about Ngrok, what is the best way for them to actually go and check it out and try it?

Peter Shafton: Certainly, Ngrok.com is easy. You will find our docs. You can just search for Ngrok SDK if you want, pick the library of language you want. They're available on GitHub. So our GitHub repo under Ngrok also has all of this code. These are open source, so you can grab 'em and look at 'em if you want, but pull 'em down and play with 'em and give us feedback, right? They've been out there, they're heavily used by a bunch of folks, but it doesn't mean there aren't places to improve and make them better. So definitely reach out, let us know, and we have support at Ngrok.com if you run into challenges and you need help, and I might even be manning that support some of those days. So you could talk to me.

Richard Moot: Excellent. Well, you might be hearing from me. If you've got a project with Square, we'd love to hear about it, reach out on Discord or X at SquareDev, don't forget to subscribe. Check out Develope.squareup.com for more. Keep building and we'll catch you next time.