AWS Morning Brief

Want to give your ears a break and read this as an article? You’re looking for this.Never miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of February 1, 2021 with Corey Quinn.

LinksForrest Brazeal article referenced: https://acloudguru.com/blog/engineering/the-lift-and-shift-shot-clock-cloud-migrationUnconventional Guide: https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/ TranscriptCorey: This episode is sponsored in part by our friends at Fairwinds. Whether you’re new to Kubernetes or have some experience under your belt, and then definitely don’t want to deal with Kubernetes, there are some things you should simply never, ever do in Kubernetes. I would say, “run it at all;” They would argue with me, and that’s okay because we’re going to argue about that. Kendall Miller, president of Fairwinds, was one of the first hires at the company and has spent the last six years the dream of disrupting infrastructure a reality while keeping his finger on the pulse of changing demands in the market, and valuable partnership opportunities. He joins senior site reliability engineer Stevie Caldwell, who supports a growing platform of microservices running on Kubernetes in AWS. I’m joining them as we all discuss what Dev and Ops teams should not do in Kubernetes if they want to get the most out of the leading container orchestrator by volume and complexity. We’re going to speak anecdotally of some Kubernetes failures and how to avoid them, and they’re going to verbally punch me in the face. Sign up now at fairwinds.com/never. That’s fairwinds.com/never.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field.Jesse: I like that. I feel like that's good. That's a solid way to start us off.Pete: Triple F. I am Pete Cheslock.Jesse: I'm Jesse DeRose.Pete: #TripleF. We should get some, I don’t know, jackets made? Mugs?Jesse: Lapel pins? I'm open. I've always wanted a Members Only jacket.Pete: If Guy Fieri can call diners, drive-ins, and dives, “Triple D,” then we can definitely call this Triple F.Jesse: We can definitely make this happen. Pete: It's not my high school transcript, either, we're talking about here. Oh, well, we are back again, continuing our series on The Unconventional Guide to Cost Management with Episode Two: the Cloud is not your data center.Jesse: Yeah, this one's gonna be a fun one. I feel like this is a topic that comes up a lot in conversations, sometimes with clients, sometimes with potential clients that are asking, “What kind of things do you see day-to-day? What are some of the big pain points that you see with your cost optimization work?” And so real quick backstory, make sure that you've listened to the previous few episodes to get some context for this segment that we're doing and get some framing for this Unconventional Guide work that we are discussing. But talking about using the Cloud as a data center, I have a lot of thoughts on this.Pete: Well, hold on a second. Isn't the Cloud just someone else's data center?Jesse: [laugh] I—yeah, you know, this is the same argument of serverless isn't actually serverless. It's just somebody else's computer.Pete: [laugh]. Someone else's Docker container. But really, there's a lot of ways we're going with this one. But we're coming at it from, obviously, a cost management perspective. And the big, bold, unpopular opinion that we're gonna say is, the most expensive way to run an application in the Cloud, is by treating the Cloud as just another data center; it's going to cost you way more than it would cost to run in a normal data center. And this goes to the world of, in the early days of Cloud, people just raging online and in conferences about the Cloud, it's so expensive. And yes, it is so expensive, if you treat it like an antiquated data center.Jesse: And really quick before you get your pitchforks out, there is this concept of ‘lift and shift’ that everybody likes to talk about or ‘technical transformation’ that everybody likes to talk about: moving from a data center into the Cloud, which a lot of people see as this movement where they just uproot everything from their local data center into AWS. And to be clear, we do recommend that. That is a solid strategy to get into the Cloud as fast as possible; just move those workloads over. But it is going to be expensive, and it's not what you ultimately want to stick with long term. So, that's ultimately the big thing to think about here. Yes, lifting and shifting from your data center into the Cloud is absolutely worthwhile. But it creates this shot clock that's now running after your migration is complete, where if you don't move on to all of the services, and opportunities, and solutions that AWS provides that are native solutions, cloud-native solutions, managed solutions, you're going to end up spending a lot more money than you want.Pete: Yeah, “The Lift And Shift Shot Clock” that was a great blog post by Forrest from ACG—ACloudGuru. We'll include a link to that in the [00:04:35 show notes]. It talks about how not only do you have technical debt accruing as you lift and shift, but potentially the brain drain as people get sick of managing this hot mess that you've lifted and shifted over. That doesn't mean you shouldn't do it. You absolutely should get into the Cloud, get into a singular vendor with your workloads as fast as possible so that you can then dedicate resources to refactoring all of that. Don't just forget about it and leave it behind. It's not going to end well for you. And you do have a time; the timer is running. So, when you're only using those core primitives—compute, object store, block store—yeah, you're going to have a pretty fixed cost on your cloud bill. But to Jesse's point, there's a lot of other services. Some of those require an engineering effort. Some of those just involve correctly using an instance type, a storage location that is more specific to its access patterns. I mean, everything is basic as T class instances—for those services that maybe don't use a lot of CPU—to reminding yourself that there are multiple tiers of S3 storage. Even Intelligent Tiering will just tier it for you. So, if you go and store everything on standard S3 storage and use GP2 volumes on EC2, yeah, it's gonna be expensive. And I know that because I look at a lot of Amazon bills, and Jesse does too, and we see the same thing. “Oh, you've got a really high bill.” “Yeah, we spend a lot on EC2.” It's, “Like, oh, let me guess. A lot of, like, I3s and C5s and M5s and a ton of EBS, right?” And they give you all this optionality, and I think it's that choice which is so overwhelming for many folks moving to the Cloud. I mean, that's, that's really the case. It's just, “What do I pick?” There's just so much.Jesse: So, let's talk about ephemerality, especially in the world of compute. Ephemerality really ...

Want to give your ears a break and read this as an article? You’re looking for this link.Never miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of January 25, 2021 with Corey Quinn.

Check out the full unconventional guide here!TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if wanting new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to AWS Morning Brief. I am Pete Cheslock.Jesse: I'm Jesse DeRose.Pete: This is Fridays From the Field. Triple F.Jesse: I feel like we've really got to go full Jean-Ralphio, Parks and Rec there. “Friday From the Feeeeeeeeeeild.”Pete: Yeah, so we're going to need to get an audio cut of that and add some techno beats to it. I think that's going to be our new intro song.Jesse: [imitates techno beats].Pete: Yeah, we're going to take both of those things. I'm glad we got this recorded because that's going to turn into a fantastic song. So, we're back to talk about The Unconventional Guide to Cost Management. And this is the first episode, this is the first of a whole slew of these that we're going to be going through from the field, these different ways that companies can impact their spend. And no, it doesn't mean go and buy the cloud management vendor of the moment to look at your spend or fire up Cost Explorer. Those are all pieces of it, but broader things, the big levers, the small levers, the levers that don't actually go back and forth, but you turn and you would have no idea because it was designed by an Amazon UX engineer.Jesse: Yeah, it's really important to call out that this discussion is looking at your cloud spend from a broader perspective and if you didn't get a chance to listen to our episode from last week, we did a little bit of an intro, framing this entire discussion. Go back and take a listen, if you haven't yet. Really talking about why looking at cloud costs through these different lenses is important. Why are you thinking about cloud cost, not just from the perspective of, “Oh, I'm going to delete these EBS snapshots,” or, “I'm going to tag all my resources,” but why is it important to think about cloud costs from other mediums?Pete: Exactly. So, don't forget, you can go to lastweekinaws.com/QA and put your questions right in that box. Your name is optional. You can just leave your name blank if you don't want anyone to know who you are. Or if you want to say something really nice about me and Jesse, and you just feel a little shy—Jesse: Aww.Pete: —that's fine, too. But just put a question in there. And we're going to dedicate some future episodes to answering those questions and diving a little deeper for those that want to know a little bit more. But as being the first episode, we got to talk about something, so what are we talking about today, Jesse? Jesse: Today we are talking about architecture and architecture context. Now, this is a really, really interesting one for me because the first thing that I think anybody thinks about when they think about cutting costs with their AWS spend is architecture decisions: something related to your infrastructure, whether that's tearing down a bunch of resources, or deleting data that's lying around. But there's a lot more to it than that context is everything. Knowing why your infrastructure is built the way it is, knowing why your application is designed the way it is, is really important to understanding your AWS cloud costs.Pete: This is where I feel like the Cloudabilitys CloudHealth, CloudCheckr Cloud-whatever companies, their products, sadly, fall down. And similar for every Amazon recommendation engine inside of AWS, they all break down. They lack the knowledge and the context of your organization. I remember a really long time ago, I had installed CloudHealth for the first time, and it said, “Hey, we've identified all these servers. They're sitting idle. Do you want us to turn them off for you?” Those servers were actually my very large Elasticsearch cluster. They were idle because if no one's querying them they don't do anything, but they sure do hold a lot of data, and they really do need to be available. So, please, please don't turn those off. But that same thing could happen if you were—you know, due to risk or compliance reasons, you had to run some infrastructure as a warm standby in another availability zone or region. Yeah, sure, it's not taking requests, it’s not doing anything, but that doesn't mean that it's not supposed to be running.Jesse: And this is really getting at one of the first big ideas, which is: work with other teams within the company. Not just other engineering teams, but product teams, possibly also security teams to understand all of the business context for your application and for your infrastructure in terms of data retention, in terms of availability, in terms of durability requirements. Because ultimately, you as a platform engineer, or an SRE, or a DevOps engineer, or whatever the hot new title is going to be a year from now, you need to understand why the infrastructure exists, and you may see servers that are sitting around idly doing nothing, but that's your disaster recovery site that is required by the business, by a service level agreement to be available at a moment's notice if something goes wrong. And so it's really important to understand what those components are and how they work together to build your overall application infrastructure.Pete: Yeah, that's a great point. I mean, having that knowledge that if you've been at a company for years, you've got a lot of this historical knowledge. People have come and gone, they've come, they've done things, they've implemented items, they've brought new features, they've gone. As companies grow may or not— may not be a single person who really truly understand the impact of various changes. I think we saw that most clearly when Amazon had their Kinesis outage: the amount of different services that were impacted was pretty large because it's just all too big for any one person to understand. But that doesn't mean that you shouldn't always continually be working to understand those different usage requirements, and chatting with the non-tech teams. Product teams, I feel like are often ignored in startups because you don't really want more work, and that's what those product teams normally do, right? But they're going to have a lot of context. I remember working in SaaS companies and looking at things like, “This? We don't use this anymore. There's no way we use this. I'm going to turn this off.” And then, I then say, well, the smarter minds prevail. I say, “Well, let me go talk to product people.” And they go, “Oh yeah. We can't get rid of that one super important API because this one client of ours paid us an obscene amou...

Want to give your ears a break and read this as an article? You’re looking for this link.SponsorsNew RelicExtraHop Never miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of January 18th, 2021 with Corey Quinn.

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.TranscriptCorey: When you think about feature flags—and you should—you should also be thinking of LaunchDarkly. LaunchDarkly is a feature management platform that lets all your teams safely deliver and control software through feature flags. By separating code deployments from feature releases at massive scale—and small scale, too—LaunchDarkly enables you to innovate faster, increase developer happiness—which is more important than you’d think—and drive transformation throughout your organization. LaunchDarkly enables teams to modernize faster. Awesome companies have used them, large, small, and everything in between. Take a look at launchdarkly.com, and tell them that I sent you. My thanks again for their sponsorship of this episode.Pete: Hello, and welcome to the AWS Morning Brief: Friday From the Field. Triple F; that's what we're calling it now. We’re going a new direction. I'm Pete Cheslock.Jesse: I'm Jesse DeRose, and I'm so excited for Triple F.Pete: Triple F. Hashtag Triple F. So, moving away, taking this into a new direction, we have… not stolen that's a little bit too aggressive. But we have been lovingly gifted this podcast from Corey Quinn after taking over while he was on paternity leave, we just kept on doing it; we never stopped, we never let him have it back. And he was nice enough just to give us this opportunity to take this Friday podcast into a new direction and talk about things that we're seeing as cloud economists in the field working with our clients.Jesse: Yeah, it really started as this confessional discussion of weird architecture patterns that we've seen, but then it definitely morphed into more of the other things that we've seen from either our work with Duckbill or work with previous engagements or previous companies. So, it just felt fitting to rebrand just ever so slightly and focus more of our efforts on what are the things that we're seeing day-to-day? What are the major problems that our clients are seeing? What are some of the pain points we've seen? What are the new features from AWS that are really the interesting and important things to talk about?Pete: Exactly. We have an interesting insight that I think a lot of folks in the industry don't get to see. We, for one, look at countless Amazon bills, seeing how people are spending their money. But we also are often reached out to directly to help engineering teams better answer questions that they're getting from finance. I mean, that's the biggest fear I have—Jesse: Yeah.Pete: —CFO comes walking over to my desk, and I haven't submitted an expense report recently like, what do they want?Jesse: [laugh]. I didn't do it. It wasn't me.Pete: Even worse is when some of your executives start learning some of these terms. And they say, “Hey, what's our cost per unit on Amazon Cloud?”Jesse: Yeah, it is something that has morphed from just a conversation about engineering teams thinking about their architecture patterns and what might be best for them to getting the entire company involved—especially finance—to ask all these questions and really think about, what's the bottom line here? How can we better understand this cloud spend?Pete: I know most people are probably thinking, “Doesn't tagging solve this problem. Can’t I just tag everything, and then I have all my answers, right?” Problem solved.Jesse: I'm sorry, did you just tell me to go F myself there, Pete?Pete: [laugh]. Obviously, we both know that even the best of companies, the most mature companies we work with, yeah, they might be about 90% plus fully tagged, but even those companies still have to put in a lot of effort to answer these questions and to understand where their spend is going. Because they say, that which gets measured gets improved. So, are you measuring your spend? Are you measuring your growth? Do you understand how your spend changes as usage changes, your customers change? I mean, there's countless questions. But there's another thing that we see, too, Jesse, right? This circle of pain, the—what is it—the cost management circle of pain.Jesse: Yeah. Yeah. It's this really fascinating idea focusing on cloud cost optimization, where a company will realize that their cloud spend has gone up for whatever reasons, and they say, “Oh, no. We need to do something about this.” Whether that is because finance has come over and asked the question, or because engineering has caught the issue. And so they go through this quick session, maybe a quarter, maybe a couple months or more of figuring out, “How can we cut costs? Can we remove resources? Can we put these practices into place? Can we build some processes? Okay, now, everything's fine, right? We've managed to bring our costs back down. We managed to get rid of all of those EBS snapshots that were collecting dust and never to be used, so now we can go about business as usual again, right?” And so then they continue on as if nothing has happened. And without making long term changes, those costs are going to rise again. And then all of a sudden, we're back in the same spot of, “Oh, no, our cloud costs have gone up, why did they go up? We did all these things to make sure that we didn't have run into this issue again. Why are our cloud costs going up again?” And the cycle just repeats. It's a really unfortunate kind of spiral.Pete: I remember my time at a startup where we were under a series of really high growth, a lot of customers coming on the platform. And my favorite meeting ever was the CEO talking about our financials. And he mentioned that our gross margin was negative 175%, which for the non-financial folks, means that for every dollar of income negative 175% is being spent for that. You normally want that number to be positive if you want to have a successful business. And remember, the line he said is, “We are going to successfully go out of business with a gross margin that is negative one hundred and seventy”—whatever I said. This is an important number that people need to think about. And what's amazing is that within a year, we had turned that around to be an extremely high gross margin because we started looking, and tracking, and bringing cultural change, and giving ownership to people to own these numbers. So, it's not just an engineering problem anymore. Everyone thinks that the Amazon bill is because your engineers built a certain thing, or turned on a certain type of instance. And sure, part of that is absolutely true, but I always like to say that your Amazo...

Special thanks to Alice Goldfuss for this week’s awesome title!Want to give your ears a break and read this as an article? You’re looking for this link.SponsorsVeeamNew RelicNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill