Links:

AWS’s Egregious Egress: https://blog.cloudflare.com/aws-egregious-egress/

Transcript

Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they’re calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you’ve come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you’re using Elasticsearch, consider not running Elasticsearch. They’re also available now in the AWS marketplace if you’d prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.

Corey: Hi there. Chief Cloud Economist Corey Quinn from the Duckbill Group here to more or less rant for a minute about something it’s been annoying the heck out of me for a while, as anyone who follows me on Twitter or subscribes to the lastweekinaws.com newsletter, or passes me in a crowded elevator will attest to, and that is AWS’s data transfer story.

Back on July 23rd—of 2021, for those listening to this in future years—CloudFlare did a blog post titled AWS’s Egregious Egress, and that was co-authored by Matthew Prince—CloudFlare’s CEO—and Nitin Rao—who is one of their employees. Presumably. That was somewhat unclear—and it effectively tears down the obnoxious—and I mean deeply obnoxious—level of AWS data transfer pricing for egress to the outside world.

And there’s a bunch of things to unpack in this blog post, where they wind up comparing AWS pricing to the wholesale bandwidth market. And they go into a whole depth for those who aren’t aware of how bandwidth is generally charged for. And the markups that they come up with for AWS are, in many cases, almost 8,000%, which is just ludicrous, in some respects, because—spoiler—every year, give or take, the wholesale cost of network bandwidth winds up dropping by about 10%, give or take. And the math that they’ve done that I’m too lazy to check, says that in effect, given that they don’t tend to reduce egress bandwidth pricing, basically ever, while the wholesale market has dropped 93%, what we pay AWS hasn’t. And that’s obnoxious.

They also talk—rather extensively—about how ingress is generally free. Now, there’s a whole list of reasons that this could be true, but let’s face it, when you’re viewing bandwidth into AWS as being free, you start to think of it that way of, “Oh, it’s bandwidth, how expensive could it possibly be?” But when you see data coming out and it charges you through the nose, you start to think that it’s purely predatory. So, it already starts off with customers not feeling super great about this. Then diving into it, of course; they’re pushing for the whole bandwidth alliance that CloudFlare spun up, and good for them; that’s great.

They have a bunch of other providers willing to play games with them and partner. Cool, I get it. It’s a sales pitch. They’re trying to more or less bully Amazon into doing the right thing here, in some ways. Great, not my actual point.

My problem is that it’s not just that data transfer is expensive in AWS land, but it’s also inscrutable because, ignoring for a second what it costs to send things to the outside world, it’s more obnoxious trying to figure out what it costs to send things inside of AWS. It ranges anywhere from free to very much not free. If you have a private subnet that’s talking to something in the public subnet that needs to go through a managed NAT gateway, whatever your transfer price is going to be has four and a half cents per gigabyte added on to it with no price breaks for volume. So, it’s very easy to wind up accidentally having some horrifyingly expensive bills for these things and not being super clear as to why. It’s very challenging to look at this and not come away with the conclusion that someone at the table is the sucker.

And, as anyone who plays poker is able to tell you, if you can’t spot the sucker, it’s you. Further—and this is the part that I wish more people paid attention to—if I’m running an AWS managed service—maybe RDS, maybe DynamoDB, maybe ElastiCache, maybe Elasticsearch—none of these things are necessarily going to be best-to-breed for the solution I’m looking at, but their replication traffic between AZs in the same region is baked into the price and you don’t pay a per-gigabyte fee for this. If you want to run something else, either run it yourself on top of EC2 instances or grab something from the AWS marketplace that a partner has provided to you. There is no pattern in which that cross-AZ replication traffic is free; you pay for every gigabyte, generally two cents a gigabyte, but that can increase significantly in some places.

Corey: I really love installing, upgrading, and fixing security agents in my cloud estate. Why do I say that? Because I sell things for a company that deploys an agent. There’s no other reason. Because let’s face it; agents can be a real headache. Well, Orca Security now gives you a single tool to detect basically every risk in your cloud environment that’s as easy to install and maintain as a smartphone app. It is agentless—or my intro would have gotten me in trouble here—but it can still see deep into your AWS workloads while guaranteeing 100% coverage. With Orca Security there are no overlooked assets, no DevOps headaches—and believe me, you will hear from those people if you cause them headaches—and no performance hits on live environment. Connect your first cloud account in minutes and see for yourself at orca dot
security. That’s orca—as in whale—dot security as in that thing your company claims to care about but doesn’t until right after it really should have.

Corey: It feels predatory, it feels anti-competitive, and you look at this and you can’t shake the feeling that somehow their network group is being evaluated on how much profit it can turn, as opposed to being the connective tissue that makes all the rest of their services work. Whenever I wind up finding someone who has an outsized data transfer bill when I’m doing the deep-dive analysis on what they have in their accounts, and I talk to them about this, they come away feeling, on some level, ripped off, and they’re not wrong. Now, if you take a look at other providers—like Oracle Cloud is a great example of this—their retail rate is about 10% of what AWS’s for the same level of traffic. In other words, get a 90% discount without signing any contract and just sign the dotted line and go with Oracle Cloud. Look, if what you’re doing is bandwidth-centric, it’s hard to turn your nose up at that, especially if you start kicking the tires and like what you see over there.

This is the Achilles heel of what happens in the world of AWS. Now, I know I’m going to wind up getting letters about this because I always tend to whenever I rant about this that no one at any significant scale is paying retail rate for AWS bandwidth. Right, but that’s ...

Advent: Peace // Aaron Bennett