Networking in the Cloud Fundamentals: Cloud and the Last Mile

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Transcript

Corey: Hello and welcome to our Networking in the Cloud, mini series sponsored by ThousandEyes. That's right. There may be just one of you, but there are a thousand eyes on a more serious note. ThousandEyes has sponsored their cloud performance benchmarking report for 2019 at the end of last year. Talking about what it looks like when you race various cloud providers. They looked at all the big cloud providers and determined what does performance look like from an end user perspective? What does the user experience look like among and between different cloud providers? To get your copy of this report, you can visit snark.cloud/realclouds. Why real clouds? Well, because they raced AWS, Azure, GCP, IBM Cloud and Alibaba, all of which are real clouds.

They did not include Oracle Cloud because once again they are real clouds. Check out your copy of the report at snark.cloud/realclouds. It's interesting that that report focuses on the end user experience because as this mini series begins to wind down, we're talking today about the last mile and its impact on what perceived cloud performance looks like. And I will admit that even having given this entire mini series and having a bit of a network engineering background, once upon a time, I still wind up in a fun world of always defaulting to blaming my local crappy ISP.

Now today, my local ISP is amazing. I use Sonic in San Francisco. I get Symmetric Gigabit. It's the exact opposite of Comcast who was my last provider until Sonic came to my neighborhood and it was fun that day because I looked up and down the block and saw no fewer than six Sonic trucks ripping Comcast out by the short and curlies. Which let's not kid ourselves, is something we all wish we could do and I was the happiest boy in town the day I got to do it. Now, the hard part is figuring out that yes, it is in fact a local ISP problem because it isn't always. This is also fortuitous because I spent the last month or so fixing my own local internet situation and today I'd like to tell you a little bit more about that as well as how and why.

Originally when I first moved into my roughly, we'll call it 2,800 square foot house, it's spread across three stories, I wound up getting EEROs, that's E-E-R-O. They're a mesh network set up that was acquired by Amazon after I'd purchased them. These are generation one. The wireless environment in San Francisco is challenging and in certain parts of my house, the reception as a result, wound up being a steaming bowl of horse crap. The big challenge was figuring out that, that's what the problem was. With weird dropouts and handoff issues, it was interesting. This one area that caused immediate improvement was not having these things talk to each other wirelessly as most full mesh systems will do, but instead making sure that they were cabled up appropriately to a switch, the central patch panel and then hooked them in. Now you have to be careful with switches because a lot of stuff won't do anything approaching full throughput because that can get expensive and a lot of consumer gear is crap.

This was a managed HP pro curved device back in the days that HP made networking equipment. That was great. And it's still crap, but it is crap that works at full line rate. So there's that. Next I wound up figuring that ... all right, it's time to take this seriously. So I did some research and talked to people I know who are actually good at things, instead of sounding on the internet like they're good at things. And I figured the next step was to buy some Ubiquiti Networks style stuff. Great. We go ahead and trot some of that out. It's an enterprise gear. It's full mesh. I of course now have a guest wifi that you have to pay for to use the hotspot. It's called Toss a coin to your wifi for an SS ID because of course it is. I have problems. And it's fun and I can play these stupid games, but suddenly every weird internet problem I had in my house started getting better as a result.

And it's astonishing how that changed my perception of various third party services. None of whom, by the way, had anything to do with my actual problem. But there were still some perceptual differences. And this impacts the cloud in a number of subtle ways and that's what I want to talk about today. So one of the biggest impacts is DNS. And I don't mean that in the sense of big cloud provider DNS, we've already talked about how DNS works in a previous episode. But rather what resolver you wind up using yourself. One of the things that I did as a part of this upgrade, is I rolled out a distribution of Linux called Pi-hole, which sounds incredibly insulting as applied to people as in, you know what, you should shut? Your Pi-hole. However, it's designed to run on top of Raspberry Pi and provide a DNS server that creatively blocks ads.

And that's super neat. I liked the idea of just blocking ad servers, but you have to trust whatever you're using for a DNS resolver because of a few specific use cases that I stumbled over as I went down this process. One, it turns out that having access to every website you'd care to visit as far as a list of things you've been doing, is not really the most privacy conscious thing in the universe. Now, for some reason, the internet collectively decided, you know who we trust with all the things that we look at on the internet and have no worries about giving that information to? That's right. Freaking Google. So eight dot eight dot eight dot eight, was a famously to remember open resolver and it works super well. It's quick. It returns everything. The problem is, is that Google's primary business model is very clearly surveillance and I don't do anything particularly interesting.

If you look at my DNS history, you're going to find a lot of things that you'd think you could use to blackmail me, but it turns out you actually can't because I talk about them on podcasts. That's right. I use Route 53 as a database. What of it? And it's all very strange just as far as even without anything to hide, I still feel this sense of pervasive creepiness at the idea that a giant company can look at this. Can look at my previous browsing history. So blocking things like that are of interest to me. So okay, instead, if I run Pi-hole that acts as my own resolver but then it winds up passing queries on to an upstream provider. I mean I could run my own, but that has other latency concerns and DNS latency when you're making requests is super indicative because the entire internet has gone collectively dumb. And decided to display a simple static webpage, You need to make 30 distinct DNS request in series and wait for them all to come back and other ridiculous nonsense that is the modern web today.

What makes this extra special is I figured out, okay, I'm not going to go with Google or CloudFlar...