AWS Morning Brief

AWS Morning Brief for the week of February 3, 2020.

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.TranscriptCorey: Hello and welcome to our Networking in the Cloud, mini series sponsored by ThousandEyes. That's right. There may be just one of you, but there are a thousand eyes on a more serious note. ThousandEyes has sponsored their cloud performance benchmarking report for 2019 at the end of last year. Talking about what it looks like when you race various cloud providers. They looked at all the big cloud providers and determined what does performance look like from an end user perspective? What does the user experience look like among and between different cloud providers? To get your copy of this report, you can visit snark.cloud/realclouds. Why real clouds? Well, because they raced AWS, Azure, GCP, IBM Cloud and Alibaba, all of which are real clouds.They did not include Oracle Cloud because once again they are real clouds. Check out your copy of the report at snark.cloud/realclouds. It's interesting that that report focuses on the end user experience because as this mini series begins to wind down, we're talking today about the last mile and its impact on what perceived cloud performance looks like. And I will admit that even having given this entire mini series and having a bit of a network engineering background, once upon a time, I still wind up in a fun world of always defaulting to blaming my local crappy ISP.Now today, my local ISP is amazing. I use Sonic in San Francisco. I get Symmetric Gigabit. It's the exact opposite of Comcast who was my last provider until Sonic came to my neighborhood and it was fun that day because I looked up and down the block and saw no fewer than six Sonic trucks ripping Comcast out by the short and curlies. Which let's not kid ourselves, is something we all wish we could do and I was the happiest boy in town the day I got to do it. Now, the hard part is figuring out that yes, it is in fact a local ISP problem because it isn't always. This is also fortuitous because I spent the last month or so fixing my own local internet situation and today I'd like to tell you a little bit more about that as well as how and why.Originally when I first moved into my roughly, we'll call it 2,800 square foot house, it's spread across three stories, I wound up getting EEROs, that's E-E-R-O. They're a mesh network set up that was acquired by Amazon after I'd purchased them. These are generation one. The wireless environment in San Francisco is challenging and in certain parts of my house, the reception as a result, wound up being a steaming bowl of horse crap. The big challenge was figuring out that, that's what the problem was. With weird dropouts and handoff issues, it was interesting. This one area that caused immediate improvement was not having these things talk to each other wirelessly as most full mesh systems will do, but instead making sure that they were cabled up appropriately to a switch, the central patch panel and then hooked them in. Now you have to be careful with switches because a lot of stuff won't do anything approaching full throughput because that can get expensive and a lot of consumer gear is crap.This was a managed HP pro curved device back in the days that HP made networking equipment. That was great. And it's still crap, but it is crap that works at full line rate. So there's that. Next I wound up figuring that ... all right, it's time to take this seriously. So I did some research and talked to people I know who are actually good at things, instead of sounding on the internet like they're good at things. And I figured the next step was to buy some Ubiquiti Networks style stuff. Great. We go ahead and trot some of that out. It's an enterprise gear. It's full mesh. I of course now have a guest wifi that you have to pay for to use the hotspot. It's called Toss a coin to your wifi for an SS ID because of course it is. I have problems. And it's fun and I can play these stupid games, but suddenly every weird internet problem I had in my house started getting better as a result.And it's astonishing how that changed my perception of various third party services. None of whom, by the way, had anything to do with my actual problem. But there were still some perceptual differences. And this impacts the cloud in a number of subtle ways and that's what I want to talk about today. So one of the biggest impacts is DNS. And I don't mean that in the sense of big cloud provider DNS, we've already talked about how DNS works in a previous episode. But rather what resolver you wind up using yourself. One of the things that I did as a part of this upgrade, is I rolled out a distribution of Linux called Pi-hole, which sounds incredibly insulting as applied to people as in, you know what, you should shut? Your Pi-hole. However, it's designed to run on top of Raspberry Pi and provide a DNS server that creatively blocks ads.And that's super neat. I liked the idea of just blocking ad servers, but you have to trust whatever you're using for a DNS resolver because of a few specific use cases that I stumbled over as I went down this process. One, it turns out that having access to every website you'd care to visit as far as a list of things you've been doing, is not really the most privacy conscious thing in the universe. Now, for some reason, the internet collectively decided, you know who we trust with all the things that we look at on the internet and have no worries about giving that information to? That's right. Freaking Google. So eight dot eight dot eight dot eight, was a famously to remember open resolver and it works super well. It's quick. It returns everything. The problem is, is that Google's primary business model is very clearly surveillance and I don't do anything particularly interesting.If you look at my DNS history, you're going to find a lot of things that you'd think you could use to blackmail me, but it turns out you actually can't because I talk about them on podcasts. That's right. I use Route 53 as a database. What of it? And it's all very strange just as far as even without anything to hide, I still feel this sense of pervasive creepiness at the idea that a giant company can look at this. Can look at my previous browsing history. So blocking things like that are of interest to me. So okay, instead, if I run Pi-hole that acts as my own resolver but then it winds up passing queries on to an upstream provider. I mean I could run my own, but that has other latency concerns and DNS latency when you're making requests is super indicative because the entire internet has gone collectively dumb. And decided to display a simple static webpage, You need to make 30 distinct DNS request in series and wait for them all to come back and other ridiculous nonsense that is the modern web today.What makes this extra special is I figured out, okay, I'm not going to go with Google or CloudFlar...

AWS Morning Brief for the week of January 27th, 2020.

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.TranscriptCorey: Welcome to the AWS Morning Briefs miniseries, Networking In the Cloud, sponsored by ThousandEyes. ThousandEyes has released their cloud performance benchmark report for 2020. They effectively race the top five cloud providers. That's AWS, Google Cloud Platform, Microsoft Azure, IBM Cloud, and Alibaba Cloud, notably not including Oracle Cloud, because it is restricted to real clouds, not law firms. It winds up being derived from an unbiased third party and metric-based perspective on cloud performance as it relates to end user experience. So this comes down to what real users see, not arbitrary benchmarks that can't be gamed. It talks about architectural and conductivity differences between those five cloud providers and how that impacts performance. It talks about AWS Global Accelerator in exhausting detail. It talks about the Great Firewall of China and what effect that has on cloud performance in that region, and it talks about why regions like Asia and Latin America experience increased network latency on certain providers. To get your copy of this fascinating and detailed report, visit snark.cloud/realclouds, because again, Oracle's not invited. That's snark.cloud/realclouds, and my thanks to ThousandEyes for their continuing sponsorship of this ridiculous podcast segment.Now, let's say you go ahead and spin up a pair of EC2 instances, and as would never happen until suddenly it does, you find that those two EC2 instances can't talk to one another. This episode of the AWS Morning Brief's Networking in the Cloud Podcast focuses on diagnosing connectivity issues in EC2. It is something that people don't have to care about until suddenly they really, really do. Let's start with our baseline premise, that we've spun up an EC2 instance, and a second EC2 instance can't talk to it. How do we go about troubleshooting our way through that process?The first thing to check, above all else, and this goes back to my grumpy Unix systems administrator days is: are both EC2 instances actually up?Yes, the console says they're up. It is certainly billing you for both of those instances, I mean, this is the cloud we're talking about, and it even says that the monitoring checks, there are two by default for each instance, are passing. That doesn't necessarily mean as much as you might hope. If you go into the EC2 console, you can validate through the system logs that they booted successfully. You can pull a screenshot out of them. If everything else was working, you could use AWS Systems Manager Session Manager, and if you'll forgive the ridiculous name, that's not a half bad way to go about getting access to an instance. It spins up a shell instance in a browser that you can poke around inside that instance within, but that may or may not get you where it needs to go. I'm assuming you're trying to connect to one of those instances or both of those instances and failing, so validate that you can get into both of those instances independently.Something else to check. Consider protocols. Very often, you may not have permitted SSH access to these things. Okay, or maybe you can't ping these and you're assuming they're down. Well, an awful lot of networks block certain types of ICMP traffic, echo requests, for example. Type eight. Otherwise, you may very well find that whatever protocol you're attempting to use isn't permitted all the way through. Note incidentally, just as an aside, that blocking all ICMP traffic is going to cause problems for your network. When things are fragmented and they need to have a different window size set for things that are being sent across the internet, ICMP traffic is how things are made aware of that. You'll see increased latency if you block all ICMP traffic, and it's very difficult to diagnose, so please, for the love of God, don't do that.Something else to consider as you go down the process of tearing apart what could possibly be going on with these EC2 instances not able to speak to each other. Try and connect to them via IP addresses rather than DNS names. Just because there's ... I'm not saying the problem is always DNS, but it usually is DNS, and this removes a whole host of different problems that could be manifesting if you just go by IP address. Suddenly resolution, timeouts, bad DNS, et cetera, fall by the wayside. When you have a system that you're trying to talk to another system and you're only using IP, suddenly there's a whole host of problems you don't have to think about. It goes well.Something else to consider in the wonderful world of AWS is network ACLs. The best practice around network ACLs is, of course, don't use them. Have an ACL that permits all traffic, and then do everything else further down the stack. The reason is that no one thinks about network ACLs when diagnosing these problems. So if this is the issue, you're going to spend a lot of time spinning around and trying to figure out what it is that's going on.The next more likely approach, and something to consider whenever you're trying to set up different ways of dividing traffic across various regimes of segmentation, is security groups. Security groups are fascinating, and the way that they interact with one another is not hugely well understood. Some people treat security groups like they did old school IP address restrictions, where anything in the following network, and you can express that in CIDR notation the way one would expect, or C-I-D-R depending on how you enjoy pronouncing or mispronouncing things, can wind up being used, sure, but you can also say members of a particular security group are themselves allowed to speak to this other thing. That, in turn, is extraordinarily useful, but it also means extremely complex things, especially when you have multiple security groups layering upon one another.Assuming that you have multiple security group rules in place, the one that allows traffic is likelier to have precedents. Note as well that there's a security group rule that is in place by default that allows all outbound traffic. If that's gotten removed, that could be a terrific reason why an instance is not able to speak to the larger internet.One thing to consider when talking about the larger internet is what ThousandEyes does other than releasing cloud benchmark performance reports. That's right. They are a monitoring company that gives a global observer perspective on the current state of the internet. If certain providers are having problems, they're well positioned to be able to figure out who that provider is, where that provider is having the issue, and how that manifests, and then present that in real time to its customers. So if you have widely dispersed users and want to keep a bit ahead of what t...

AWS Morning Brief for the week of January 20th, 2020.

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.TranscriptCorey: Welcome to the AWS Morning Brief, specifically our 12-part mini series, Networking In The Cloud, sponsored by ThousandEyes. ThousandEyes recently released their state of the cloud benchmark performance report. They raced five clouds together and gave a comparative view of the networking strengths, weaknesses, and approaches of those various providers. Take a look at what it means for you. There's actionable advice hidden within, as well as incredibly useful comparative data, so you can start comparing apples to oranges instead of apples to baseballs. Check them out and get your copy today at snark.cloud/realclouds. That's snark.cloud/realclouds because Oracle cloud was not invited to participate.Now, one thing that they did not bother to talk about in that report, is how much all of that data transfer across different providers costs. Today I'd like to talk about that, which is a bit of a lie because I'm not here to talk about it at all, I'm here to rant like a freaking lunatic for which I make no apologies whatsoever.This episode is about data transfer pricing in AWS. Because honestly I need to rant about something and this topic is entirely too near and dear to my heart, given that I spend most of my time fixing AWS bills for interesting and various sophisticated clients.Let's begin with a simple question. The answer to which is guaranteed to piss you off like almost nothing else. What does it cost to move a gigabyte of data in AWS? Think about that for a second. The correct answer, of course, is that nobody freaking knows. There is no way to get a deterministic answer to that question without asking a giant boatload of other questions.Let me give you some examples, and before I do, I would like to call out that every number I'm about to mention applies only to us-east-1, because of course different regions in different places have varying costs, that every single one of these numbers is different in other places sometimes, but not always. Why? Because things are awful. I told you I was going to rant. I'm not apologizing for it at this point.Let's begin simply and talk about what it takes to just shove a gigabyte of data into AWS. Now in most cases that's free. Inbound bandwidth is always free to AWS usually, until it passes through with load balancer or does something else but we'll get there. What does it cost to move data between two AWS regions? Great. The answer to that is, two cents per gigabyte in the primary regions, except there's one use case which gets slightly less. And that is moving between us-east-1 and us-east-2. One is in Virginia, two is in Ohio. That is half price at one cent per gigabyte. My working theory behind that is that it's because even data wants to get the hell out of Ohio.Let's take it a step further. Let's say you were in an individual region. What does it cost to move data from 1-AZ to another? The documentation was exquisitely unclear, and I had to do some experiments with spinning up a few instances in otherwise empty AWS accounts, and using DD and Netcat to hurl data across various links to find out the answer and then wait till it showed up on my bill. The answer is it also costs 2 cents per gigabyte, the same cost as region to region. It's one cent per gigabyte out of an AZ and one cent per gigabyte in to an AZ. And that's right, it means you get charged twice. If you move 10 gigabytes, you are charged for 20 gigabytes on that particular metric.This also has the fun ancillary side effect of meaning that moving data between Virginia and Ohio is cheaper to do that cross region transfer than it is to move that same data within an existing region. Oh wait, it gets dumber than that. What do load balancer data transfer fees look like? The correct answer is who the hell knows? On the old classic load balancers, it was 0.8 cents per gigabyte in or out to the internet and there was also an instance fee, but that's not what we're talking about today. Traffic from any existing load balancer today to something inside of an AZ is free unless it crosses an availability zone and then we're back into cross AZ data transfer territory and anything going from an availability zone to a load balancer costs one cent per gigabyte.Now the newer load balancer generations, the ALDs and the NLDS, what does that cost? Nobody freaking knows because data throughput is just one of several dimensions that go into a load balancer capacity unit, which mean that what your data transfer price is going to look like is going to vary wildly because in this particular case, it's not data transfer itself. There's still that as it leaves, but you also have to pay for this as an additional through the load balancer fee, but it's blended into an LCU, so it's not at all obvious at times that that is in fact what you're being billed for.In another episode of this mini series, we talked about global accelerator. Now there's a site to site VPN option, which they had for a while, but at re:Invent last year they announced a accelerated VPN option that leverages a lot of global accelerator technology to let that site to site VPN take advantage significantly of the global accelerator. Now what does that cost? I could not freaking tell you. There are, I am not exaggerating, five distinct billing line items, if you run an accelerated site to site VPN and of course, all of them cost you money. I am not exaggerating. That is the actual state of the world. It is incredibly annoying. It is so annoying that I'm going to have to take a break before I blow a blood vessel to tell you more about ThousandEyes instead.So other than the cloud report, what is ThousandEyes? They effectively act as the global observer that watches the entire internet from a whole bunch of different listening posts around that internet and keeps track in near real time of what's going on, what's being slow, what providers are having issues and giving information directly to your folks on your side to be able to understand, adapt and mitigate those outages and slow downs. It helps immediately get to the point of is this a networking problem globally or is it our last crappy code deploy that broke things? If this sounds like something that might be useful for you or your team, I encourage you to check them out at thousandeyes.com. They're a fantastic company with a fantastic product and best of all their billing makes sense.We're back to ranting again. That's right. My problem with the AWS data transfer pricing is not that it's shitty and complex, but also that it's expensive. Pricing largely has not changed since AWS...

AWS Morning Brief for the week of January 13th, 2020.

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.TranscriptCorey: Welcome back to Networking In The Cloud, a special 12 week mini feature of the AWS morning brief sponsored by ThousandEyes. This week's topic, The Cloud in China, but first, let's talk a little bit about ThousandEyes. You can think of ThousandEyes as the Google maps of the internet, just like you wouldn't leave San Jose to drive to San Francisco without checking which freeway to take because local references are always going to resonate the best when telling these stories, business rely on ThousandEyes to see the end to end paths that their applications and services are taking from their servers to their end users, to identify where the slowdowns are, where the pile ups are, and what's causing these issues. They can use ThousandEyes to figure out what's breaking and ideally notify providers before their customers notice. To learn more, visit thousandeyes.com. And my thanks to them for their sponsoring of this mini series.Now, when we're talking about China, I want to start by saying that I'm not here to pass judgment. Here in the United States, we're sort of the Oracle cloud of foreign policy, so Lord knows that my hands aren't clean any. Instead, I want to have a factual discussion about what networking in China looks like in the world of cloud in 2020. To start, China is a huge market. The market for cloud services in China this year is expected to reach just over a hundred billion dollars. So there's a lot of money on the table, there's a lot riding on companies making significant inroads into an extremely lucrative market that is extremely technologically savvy.Historically, according to multiple Chinese cloud executives who were interviewed for a variety of articles, China's enterprise IT market is probably somewhere between five to seven years behind most Western markets. That means that there's a huge amount of opportunity for companies to be able to make inroads and make an impact on that market before it winds up being dominated, like a lot of the Western markets have been by certain large Seattle-based cloud providers, ahem, ahem.Now, due to Chinese regulations, in order to run a cloud provider in China, it has to be operated by a Chinese company. That's why Microsoft works with a company called 21Vianet, whereas AWS has two partners, Beijing Sinnet and NWCD. Those local partners in fact own and operate the physical infrastructure that the cloud providers are building in China and become known as the seller of record. Although the US cloud companies of course do, or at least ostensibly retain all the rights to their intellectual property, either trademarks, their copyrights, etc.That said, if you take a look at any of the large cloud providers, service and region availability tables, there's very clearly a significant lag between when services get released in most regions and when they do inside the mainland China regions. Some of the concern, at least according to people off the record, comes down to concern over intellectual property theft. And in the current political climate where we have basically picked an unprovoked trade war with China, it winds up complicating this somewhat heavily. If for no other reason, then companies are extremely skittish about subjecting what they rightly perceive to be their incredibly valuable intellectual property to the risks of operating inside of mainland China, so on the one hand they don't want to deal with that. On the other, there are over half a billion people in China with smartphones, just shy of 900 million people on the internet in one form or another. So there's an awful lot of money at stake. So companies find themselves rather willing to overlook some things that they otherwise would not want to bother with. Now again, I'm not here to moralize, I just find the idea to be somewhat fascinating.Most of that stuff you can find out just from reading news articles and various press releases. So let's go a little bit further into how companies are servicing the Chinese market. Not for nothing, but picking on AWS because they are the incumbent in this space, and this is the AWS morning brief. But looking at the map on my wall, they have regions in Tokyo, in Seoul, in Hong Kong, in Singapore and Mumbai. If you squint enough, that sort of forms a periphery around the outside of mainland China. Here in the real world, if it's at all feasible, companies tend to use those regions that are more or less scattered around China, rather than within China if it is even slightly feasible and then provide services to their customers inside of China through those geographically local regions without having to deal with having a physical presence inside of China. You can learn a lot about this by looking at ThousandEyes 2019 Public Cloud Performance Benchmark Report, where they wound up figuring out what's going on with IBM, AWS, Azure and Google Cloud, and of course Alibaba this year, which is interesting and we'll get there in a minute because this is restricted to real clouds.Oracle cloud is not a real cloud and thus was not invited. Figure out what the different architectural conductivity differences are between these cloud providers. Take a look at the AWS global accelerator and how it pans out and what you can actually expect from real world networks going to other real world networks, and see what it is that makes sense for various use cases. My thanks again to ThousandEyes for sponsoring this podcast. You can get your own copy of the report at snark.cloud/real clouds, that's snark.cloud/realclouds.One of those real clouds as mentioned is Alibaba. The reason that I bring them up is that they currently dominate China's cloud market. Alibaba has something on the order of a 43% market share inside of mainland China. Second behind them with 17.4% is 10 Cent. 10 Cent is also growing rapidly. AWS is up there as well, given their significant posture and other places. But then there's a whole smattering of small scale cloud operators that are still vying for a piece of a very large, very lucrative pie.Now, if you're talking to any of those providers inside of China, then the networking works pretty much like you'd expect it to anywhere else on the planet. The challenge and why this is worth an entire episode is what happens when you try to network outside of China into the rest of the internet. Let's talk a little bit about China's great firewall. This was started roughly in 1998 in order to enforce Chinese law. News, shopping sites, stereo search engines and pornography are all blocked through a wide variety of methods in accordance with Chinese law, that tends to change and ebb and flow. No...

AWS Morning Brief for the week of January 6th, 2020.

AWS Morning Brief for the week of December 30th, 2019.