AWS Morning Brief

AWS Morning Brief for the week of August 16, 2021 with Corey Quinn.

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/re:imagining-aws-re:invent Never miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of August 9 2021 with Corey Quinn.

Links:AWS’s Egregious Egress: https://blog.cloudflare.com/aws-egregious-egress/TranscriptCorey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they’re calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you’ve come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you’re using Elasticsearch, consider not running Elasticsearch. They’re also available now in the AWS marketplace if you’d prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: Hi there. Chief Cloud Economist Corey Quinn from the Duckbill Group here to more or less rant for a minute about something it’s been annoying the heck out of me for a while, as anyone who follows me on Twitter or subscribes to the lastweekinaws.com newsletter, or passes me in a crowded elevator will attest to, and that is AWS’s data transfer story.Back on July 23rd—of 2021, for those listening to this in future years—CloudFlare did a blog post titled AWS’s Egregious Egress, and that was co-authored by Matthew Prince—CloudFlare’s CEO—and Nitin Rao—who is one of their employees. Presumably. That was somewhat unclear—and it effectively tears down the obnoxious—and I mean deeply obnoxious—level of AWS data transfer pricing for egress to the outside world.And there’s a bunch of things to unpack in this blog post, where they wind up comparing AWS pricing to the wholesale bandwidth market. And they go into a whole depth for those who aren’t aware of how bandwidth is generally charged for. And the markups that they come up with for AWS are, in many cases, almost 8,000%, which is just ludicrous, in some respects, because—spoiler—every year, give or take, the wholesale cost of network bandwidth winds up dropping by about 10%, give or take. And the math that they’ve done that I’m too lazy to check, says that in effect, given that they don’t tend to reduce egress bandwidth pricing, basically ever, while the wholesale market has dropped 93%, what we pay AWS hasn’t. And that’s obnoxious.They also talk—rather extensively—about how ingress is generally free. Now, there’s a whole list of reasons that this could be true, but let’s face it, when you’re viewing bandwidth into AWS as being free, you start to think of it that way of, “Oh, it’s bandwidth, how expensive could it possibly be?” But when you see data coming out and it charges you through the nose, you start to think that it’s purely predatory. So, it already starts off with customers not feeling super great about this. Then diving into it, of course; they’re pushing for the whole bandwidth alliance that CloudFlare spun up, and good for them; that’s great.They have a bunch of other providers willing to play games with them and partner. Cool, I get it. It’s a sales pitch. They’re trying to more or less bully Amazon into doing the right thing here, in some ways. Great, not my actual point.My problem is that it’s not just that data transfer is expensive in AWS land, but it’s also inscrutable because, ignoring for a second what it costs to send things to the outside world, it’s more obnoxious trying to figure out what it costs to send things inside of AWS. It ranges anywhere from free to very much not free. If you have a private subnet that’s talking to something in the public subnet that needs to go through a managed NAT gateway, whatever your transfer price is going to be has four and a half cents per gigabyte added on to it with no price breaks for volume. So, it’s very easy to wind up accidentally having some horrifyingly expensive bills for these things and not being super clear as to why. It’s very challenging to look at this and not come away with the conclusion that someone at the table is the sucker.And, as anyone who plays poker is able to tell you, if you can’t spot the sucker, it’s you. Further—and this is the part that I wish more people paid attention to—if I’m running an AWS managed service—maybe RDS, maybe DynamoDB, maybe ElastiCache, maybe Elasticsearch—none of these things are necessarily going to be best-to-breed for the solution I’m looking at, but their replication traffic between AZs in the same region is baked into the price and you don’t pay a per-gigabyte fee for this. If you want to run something else, either run it yourself on top of EC2 instances or grab something from the AWS marketplace that a partner has provided to you. There is no pattern in which that cross-AZ replication traffic is free; you pay for every gigabyte, generally two cents a gigabyte, but that can increase significantly in some places.Corey: I really love installing, upgrading, and fixing security agents in my cloud estate. Why do I say that? Because I sell things for a company that deploys an agent. There’s no other reason. Because let’s face it; agents can be a real headache. Well, Orca Security now gives you a single tool to detect basically every risk in your cloud environment that’s as easy to install and maintain as a smartphone app. It is agentless—or my intro would have gotten me in trouble here—but it can still see deep into your AWS workloads while guaranteeing 100% coverage. With Orca Security there are no overlooked assets, no DevOps headaches—and believe me, you will hear from those people if you cause them headaches—and no performance hits on live environment. Connect your first cloud account in minutes and see for yourself at orca dot security. That’s orca—as in whale—dot security as in that thing your company claims to care about but doesn’t until right after it really should have.Corey: It feels predatory, it feels anti-competitive, and you look at this and you can’t shake the feeling that somehow their network group is being evaluated on how much profit it can turn, as opposed to being the connective tissue that makes all the rest of their services work. Whenever I wind up finding someone who has an outsized data transfer bill when I’m doing the deep-dive analysis on what they have in their accounts, and I talk to them about this, they come away feeling, on some level, ripped off, and they’re not wrong. Now, if you take a look at other providers—like Oracle Cloud is a great example of this—their retail rate is about 10% of what AWS’s for the same level of traffic. In other words, get a 90% discount without signing any contract and just sign the dotted line and go with Oracle Cloud. Look, if what you’re doing is bandwidth-centric, it’s hard to turn your nose up at that, especially if you start kicking the tires and like what you see over there.This is the Achilles heel of what happens in the world of AWS. Now, I know I’m going to wind up getting letters about this because I always tend to whenever I rant about this that no one at any significant scale is paying retail rate for AWS bandwidth. Right, but that’s ...

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/The-Clouds-Competing-Approaches-to-DeprecationNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of August 2, 2021, with Corey Quinn.

Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they’re calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you’ve come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you’re using Elasticsearch, consider not running Elasticsearch. They’re also available now in the AWS marketplace if you’d prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.Amy: I’m Amy Negrette.Tim: And I’m Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways that we’ve seen AWS used and abused in the wild. Today, we’re going to be talking about the relationship between cost optimization work and investing in reservations or private pricing with AWS. This is kind of a situation conversation. Let’s say you’ve got three months left on your EDP, or maybe your spend is reaching the point where you’re starting to think about investing in, or signing an EDP. But you’ve also got some cost optimization opportunities that you want to work on. How do you prioritize those two ideas?Tim: I think when we’re talking about this, first it’s important to talk about what goes into an EDP, like, what it is and what it involves. So, EDP for AWS is Enterprise Discount Program, and what it involves is you making a monetary commitment to AWS to spend a certain amount over a certain amount of time. So, a three year EDP, you’re going to spend X amount in one year, X amount the next year, and X amount the third year for a total of whatever you decide on. So, you know, AWS typically going to want 20% year-over-year growth, so you’re going to say—you’re going to spend a million dollars, and then a million dollars plus 20% is something like $1.2 million; then, you know, 20% of that and so forth and so on.And then so your total commit will be somewhere around, like, $3.6, $3.7 million, we’ll say, right? Once you signed the EDP, that’s how much you’re going to get billed for, minimum. So, it’s important to cost optimize before you make that commitment because if AWS is expecting you and you’re on the hook to make 20% year-over-year growth, but then you optimize and you save 20% of your bill, it won’t matter because you’re still going to owe AWS the same amount of money even if you cost-optimize.Jesse: Yeah, I want to take a step back and talk about EDP—as we mentioned, Enterprise Discount Program—also has—there’s a couple other flavors that give you a variety of different types of discounts. EDP generally focuses on a cross-service discount for a certain annual commit, but there are also private pricing agreements or private pricing addendums, and other private pricing, generally speaking, offered by AWS. All of those basically expect some amount of either spend on a yearly basis or some amount of usage on a yearly basis, in exchange for discounts on that usage. And really, that is something that, broadly speaking, we do recommend you focus on, we do recommend that you invest in those reservations, but it is important to think about that—I agree—I would say after cost optimization work.Amy: The thing is that AWS also provides discounts that are commandment required, that you don’t need an EDP for, namely in reservations and savings plans. So, you would similarly be on the hook if you decide, “I have this much traffic, and I want to savings plan or reservation for it.” And then suddenly you don’t have that requirement anymore, but you still have to make up that commitment.Tim: I’ll say, I think too, that also matters when you’re looking at things like reservations. If you’re going to reserve instances, you’re going to get an idea of how many you’re specifically going to need, so that way you’re not reserving too many, and then you optimize, you downsize, and all of a sudden, now you have all these reservations that you’re not going to use.Jesse: One thing to also call out: when renewing an EDP, or private pricing, or when entering into a new agreement for any kind of private pricing with AWS, they will generally look at the last six months of your usage—either broadly speaking if it’s an EDP, or specifically within a specific AWS service if it’s private pricing for a specific service—and they will double, basically, that spend over the last six months and expect you to continue spending that. So, if you spent a high amount of money over the last six months, they’re going to expect that kind of trend to continue, and if you enter into an agreement with that 12-month spend, essentially, going forward, and then make cost optimization changes, you’re ultimately going to be on the hook for this higher level of spending you’re not spending any more. So, if you focus on that cost optimization work first, it will ultimately give you the opportunity to approach AWS with a lower commit level, which may ultimately mean a lower tier of percentage discount, but ultimately, then you’re not on the hook for spend that you wouldn’t otherwise be spending.Tim: I think one of the main things people see, too, is when they’ve looked at, like, oh, what’s the low hanging fruit for me to get lower the cost? They’ll think, “Oh, well, I can do EDP,” because AWS is going to want you to sign on; they would love to have that guaranteed money, right? And a lot of times, that’s going to be a much easier thing to do, organizationally, than the work of cost optimization because almost always, that involves engineering hours, it involves planning, it involves some changes that are going to have to be made that’s probably going to be harder than just signing a contract. But again, it’s super necessary because you really need to know, have eyes open, when you’re going to go, and figure out what you’re going to commit, whether it’s private pricing agreement, or an EDP, or reservations. You want to go in there and at least decide what you want to do, what it should look like, get as optimized and as lean as you can, then make your commitments. And then once you get to an EDP, that’s when you’re going to want to do your reservation or savings plans purchases and things like that, so you do that with a discount across those.Jesse: Yeah, that’s another important thing to point out: focus on the cost optimization work first. Get your architecture, your workloads, as optimized as possible, or as optimized as you can within the given timeframe, then focus on the investment because then you’ll be able to have a much better idea of what your growth is going to look like year-over-year for an EDP or any kind of private pricing. And then after that, purchase any reservations, like reserved instances or savings plans because ultimately, then you get not only the discount from the EDP that you just signed, but any upfront payments that you make, or partial upfront payments that you make for those reservations applied towards your first year EDP. So ultimately, not only are you getting a discount on that, but you are also able to...

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/the-amazonian-evil-infecting-aws Never miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of July 26, 2021 with Corey Quinn.

TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.Amy: I’m Amy Negrette.Tim: And I’m Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways we’ve seen AWS used and abused in the wild. Today, we’re going to be talking about AWS, an open-source software. Now, that’s kind of a broad topic, but there have been some specific, recent events I’ll say, over the last year maybe or maybe even less, related to AWS and open-source software that really got us talking, and I wanted to have a deeper conversation with both of you on this topic.Tim: Well, you should probably start by going over some of the things that you’re mentioning, when you say ‘some of these things,’ what are those things, Jesse?Jesse: Yeah. So, I think the best place to start is what constitutes open-source software. And specifically, I think, not just what constitutes open-source software, but how does that differ from an open-source company?Tim: So, open-source software can be anything: Linux kernel, bash, anything like that, any Python functioning module. If you make a piece of software, whatever it is, and you license it with one of the various open-source licenses, or your own open-source license or whatever, it’s something that the community kind of owns. So, when they get big, they have maintainers, everything like that, but at its essence, it’s a piece of software that you can freely download and use, and then you’re free to modify it as you need, and then it’s up to the specifics of the license to whether you’re required to send those modifications back, to include them, or to whatever. But the essence is that it’s a piece of software that’s free for me to use and free for me to modify under it’s license.Jesse: And one of the other things I want to add to that is, correct me if I’m wrong here, but isn’t a lot of open-source software is very community-owned, so there’s a lot of focus on folks from the community that is using this software giving back not because they need to under the licensing, necessarily, but because they want to continue using this and making it better over time.Amy: I think one of the issues is that becomes a very opinionated kind of statement where there are a lot of people in the open-source community who feel that if you’re going to use something and make changes to better suit what your needs are, that you should be able to submit those changes back to the community, or back to whoever owns the base of the software. But that said, it’s like the community edition of MySQL before Microsoft bought it, where the assumption was that there’s essentially a candidate of it that anyone can use without the expectation of submitting it back.Jesse: So, that’s a broad definition of open-source software, but how does open-source software, broadly speaking, differ from an open-source company? I’m thinking specifically there is the open-source software of Elasticsearch, for example, or I should say, previously the open-source software of Elasticsearch that was owned by the open-source company, Elastic. So, what does that relationship look like? How does an open-source company like that differ from the open-source software itself?Tim: So, there are typically a couple of ways. Usually, a company that is the owner of an open-source product still has some kind of retention of the IP in their various licenses that they can do that with, but essentially—and this is in the words of one of the founders of Elastic—that they’re benevolent dictators over the software. And so they allow folks to contribute, but they don’t have to. And most of those open-source software companies will have a commercial version of that software that has other features that are not available, packages with support or some of the things like that, some kind of value-added thing that you’re going to wind up paying for. The best way to describe—like you said—there’s the company Elastic and then the product Elasticsearch.I relate back to before: there was Red Hat Linux, which was open-source, and then the company Red Hat. And I remember when they went public and everyone was shocked that a company can make profit off of something they gave away for free. But while the core of the software itself was free, the support was not free, nor was the add-on features that enterprises wanted. And so that tends to be kind of what the business model is, is that you create the software, it’s open-source for a while to get a big user base, and then when it gets adopted by enterprises or people that really would pay for support or for other features, that’s when the license tends to change, or there’s a fork between the open-source version and then the commercial version.Jesse: And it definitely sounds like there can be benefits to an open-source company essentially charging for not just the open-source software, but these extra benefits like supports and additional features because I know I’ve traced multiple code bugs back to a piece of open-source software that there’s a PR or an issue that has been sitting open for months, if not longer because the community just doesn’t have the time to look into the issue, doesn’t have the time to work on the issue, they are managing it on their own, separate as a side job, separate from their day-to-day work. Whereas if that is a bug that I’m tracing back to a feature in an open-source piece of software, or I should say software that I am paying for through an open-source company, I have a much clearer support path to a resolution to resolving that issue.Tim: And I think what the end up doing is then you see it more like a traditional core software model, like, you know, a la Oracle, or something like that where you pay for the software essentially, but it comes packaged with these things that you get because of it, and then there’s a support contract on top of it, and then there’s hosting or cloud, whatever it is, on top of that, now, but you would still end up paying for the software and then support as part of the same deal. But as you know, these are for-profit companies. People get paid for them; they are publicly traded; they sell this software; they sell this product, whether it’s the services or the hosting, for profit. That is not open-source software. So, if company X that makes software X, goes under, they are acting like the software would then go under as if the software doesn’t belong to the community.So, a business that goes after a business is always going to be fair play; I believe they call it capitalism. But when you talk about going after open-source software, you’re looking at what Microsoft was doing in the ’90s and early 2000s, with Linux and other open-source challenges to the Windows and the other paid commercial enterprise software market. When folks started using Linux and servers because it was free, c...