AWS Morning Brief

Links:Unconventional GuideTranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I'm Pete Cheslock.Jesse: I'm Jesse DeRose.Pete: We're back again. Hashtag Triple F.Jesse: It's going to be a thing.Pete: We're still trying to make it a thing. Desperately trying to make it a thing. Otherwise, we're just going to look like fools, Jesse, if it's not a thing.Jesse: Oh now, I wouldn't want to look like a fool, you know, next to anybody else in my company.Pete: [laugh]. It definitely seems to be the one that trait you need to have to work at Duckbill is, to be okay looking like a fool. So, we are midway through the Unconventional Guide to AWS Cost Optimizations, cost savings. And we have been sharing a link on pretty much if not all of these recordings where you can send us feedback. And you can send us questions. And someone finally sent us a question. I think people are listening out there, Jesse. Isn't that great? Jesse: We have one follower. Yay.Pete: It's amazing. So, we are really happy that someone asked us a question. You can be the next person to ask us a question by going into lastweekinaws.com/QA. That's not our quality assurance site for testing, new branding things, and new products. QA is for ‘question and answer.’ So, go there, check it out, drop in a message, you can put your name there or not, it's totally fine. But this first question—well, first, I need to actually, I need to admit something. I'm lying right now. This question actually came in months ago. We saw it and thought that was a great question, we should answer it at some point. And then we forgot about it. So we're bringing it back up again, and I think it's relevant so I don't feel too bad about it.Jesse: Yeah, we saw this question around the time that we started recording the entire Unconventional Guide series. And apologies to this listener. This is a very good question. We want to talk about it, so we are talking about it today. But it took a little bit of a time for us to get to this. Pete: But you know what? We made it. We're here.Jesse: We’re here.Pete: We're here. So, Nick Moore asked this great question. He said, “Hey, Pete and Jesse. Very much enjoying your Friday segment on the Morning Brief.” Thank you very much for that. “If possible, I'd like to hear you talk about your experiences with cost optimization for quote, ‘big data’ projects in the cloud, i.e. Using platforms like Hadoop to process large and complex data, either using pass—like, EMR or [IS 00:03:03]. Is this something that your customers ask about often/at all? And how do or would you approach it? Thanks, again.” Well, hey, this is a truly awesome question. And at a high level, many of our clients actually are pretty heavy users of various Amazon services for their, kind of, big data needs. And big data, it's all relative, right? I mean, to some companies, big data is in the hundreds of terabytes, to other companies it's in the hundreds of petabytes. It's totally relative, but at the end of the day, it's going to be a challenge, no matter how big of a company you are. Your big data challenges are always a challenge.Jesse: You've got some kind of data science or data analytics work that you want to do with large data sets. That may be large datasets comparatively to the work that you're doing; that may be large data sets comparatively to the industry. Doesn't matter. Either way, it is big data projects, and there are many, many, many, many solutions out there.Pete: What's interesting, too, is I think the reason that this has grown in prevalence over the last year, more of our clients have been using more of these services is simply because the barrier to entry on these projects, on these engagements, is so low. You can get started on Amazon with some Athena and Glue, maybe some EMR, for just an incredibly low cost. And also, from a technical standpoint, it's not that challenging. I mean, as a good example, most reasonably technical people could take their cost and usage report, get it integrated into Athena using AWS Glue in minutes. I mean, without using CloudFormation. I mean just clicking through to set it up. And honestly, for some clients, their cost and usage reports, and that's a big data problem. That could be—if you're not storing it in Parquet, if you're actually storing it in CSV because you're a mad person, those could be hundreds of gigabytes a day in volume.Jesse: Yeah. So, when we talk about big data tasks, there's a couple different services that we generally see folks using within AWS. We generally see S3, Kinesis, and most obviously, EMR.Pete: Yeah, exactly. And we're seeing new services like Kinesis, expanding on Kinesis: Kinesis Firehose, when that came out; people are using that for some of their big data needs, especially when trying to stream data into S3. That's a really powerful feature that Firehose can do. And then, once it's in S3, the question that our clients often ask is, kind of, “What do I do with it now?” And if we dive into just S3, and you've got your data in S3, where are the kinds of places that we see unnecessary charges for data warehouse tasks? Jesse: Honestly, it's unfortunately kind of both of the major places that you're going to be charged for S3 which is, for your storage costs, and for your requests. Pete: So, what you're saying is that all S3 charges are unnecessary. [laugh].Jesse: Just get rid of it. Just put all that on EBS volume somewhere. Turn off your S3, you're solid. Pete: Exactly. It is kind of funny, but it's true. I mean, there's ways to abuse both of those pricing models, whether it's storage or requests. The first place that we honestly see a lot of this is just people are data pack rats. And let's be honest; I'm one of them as well, I have a NAS setup at home with, like, 30 terabytes of hard drives on it. I don't throw anything away digitally. Turns out most of our other clients are the exact same way, and sadly, a lot of them use standard storage for S3, which we talk about often. It's common: you get started with the standard storage, that's a great place. But for big data tasks, it's often the wrong storage solution, especially for data that maybe has already been transformed and is stored in a more efficient format; maybe it's queried infrequently. There's two ways to so...

Want to give your ears a break and read this as an article? You’re looking for this link.Never miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of February 8, 2021 with Corey Quinn.

TranscriptCorey: This episode is sponsored in part by our friends at Fairwinds. Whether you’re new to Kubernetes or have some experience under your belt, and then definitely don’t want to deal with Kubernetes, there are some things you should simply never, ever do in Kubernetes. I would say, “run it at all.” They would argue with me, and that’s okay because we’re going to argue about that. Kendall Miller, president of Fairwinds, was one of the first hires at the company and has spent the last six years the dream of disrupting infrastructure a reality while keeping his finger on the pulse of changing demands in the market, and valuable partnership opportunities. He joins senior site reliability engineer Stevie Caldwell, who supports a growing platform of microservices running on Kubernetes in AWS. I’m joining them as we all discuss what Dev and Ops teams should not do in Kubernetes if they want to get the most out of the leading container orchestrator by volume and complexity. We’re going to speak anecdotally of some Kubernetes failures and how to avoid them, and they’re going to verbally punch me in the face. Sign up now at fairwinds.com/never. That’s fairwinds.com/never.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I am Pete Cheslock.Jesse: I'm still Jesse DeRose.Pete: We're still here. And you can also be here by sending us your questions at lastweekinaws.com/QA. We're continuing our Unconventional Guide to AWS Cost Management series, and today we're talking about moving data. It's not cheap, is it?Jesse: No, it's definitely not cheap. It is expensive, and it's painful. And we're going to talk about why, today. And a reminder, if you haven't listened to some of the other episodes in this series, please go back and do so. Lots of really great information before this one and lots of really great information coming after this one. I'm really excited to dive in.Pete: Yeah, look, they're all great episodes in the end of the day, right? They're just all fantastic.Jesse: Yeah.Pete: If I do say so myself.Jesse: All of the information is important; all of the information is individually important—I think that's probably the best way to put it. You can listen to all these episodes and implement maybe just a handful of things that work best for you; you can listen to all these episodes and implement all of them, all of the suggestions. There's lots of opportunities here.Pete: If you do actually go and implement all of these suggestions, you really should go to lastweekinaws.com/QA and tell us about it. We'd be very curious to hear how it goes. But if you're struggling with any of these, just let us know as well. These are things that are measured in long periods of time. It is rare that we run into engagements with clients that you can just click box, save money. Now, don't get me wrong; there's a whole bunch of those, too. But if you want to just fundamentally improve how you're using the Cloud and how you're saving money, those projects are multi-year investments. It's just all of this stuff takes a long time. And you just got to manage those expectations appropriately. And specifically around this topic, moving data, it is—as Jesse said—painful. It is expensive, especially in Amazon. They will charge you to move the tiniest bit of data literally everywhere, with, like, two minor exceptions. And it's just the worst. Data storage costs, so Duckbill Group, we've kind of become these experts on data transfer and data storage costs, understanding just the complexity around them. And I feel like a lot of times folks only think about the storage being the biggest driver of their spend. Jesse: Absolutely.Pete: You know, you never delete your data. But you put it all on S3, right, Jesse? Like that's a cheap place to put your data.Jesse: Absolutely. Worthwhile. Put it in S3 standard storage, call it a day. I'm done, right? Pete: Yeah, just do my little, like, wipe my hands, and go on, and we're good. Most people put it in standard storage, just like most people use gp2 EBS volumes; that's the standard everything. And that could be a big driver of cost, but more likely the larger driver—because it's a little bit more hidden, it's a little bit more spread around your entire bill is the transferring of data, the moving data around. And I say moving specifically because there are some services that are charged via I/Os. Via actually putting data into it or taking data out, not just the data transfer.Jesse: I think it's also really important to call out that most companies that move into the Cloud don't realize that data transfer is something that AWS will charge you for, so I want to make that explicitly clear. As Pete mentioned, in almost every case moving data around, AWS will charge you for that versus in a data center environment where that's kind of hidden, that's not really explicitly a line item in your bill. And here, it absolutely is a line item in your bill and absolutely should be thought of as an important component to optimize. Pete: Exactly. In the data center world, for any of the folks out there that are in a data-center land, or maybe hybrid-cloud land, your networking costs are, I mean, it's largely a sunk cost. You've got your switches and your lines that run, maybe you're—get charged for the cross-connects, and interacting, data transferring to other areas and things like that. But within your racks, within your own secure domains, you don't have to really think about the cost of those network communications because it's already paid for. And you're definitely not charged at a per-gigabyte level like you are on Amazon.Jesse: So, we talked about this a little bit before in a previous episode, when we talked about context is king. Context for your application infrastructure is really, really important; understanding how your application interacts with other applications within your cloud infrastructure ecosystem; how your data moves between workloads. All of these things are really, really important, and so specifically, when we talk about data transfer, it's really important to not just understand how your data is moved around, but why your data is moved around. So, we really like to suggest working with all of the teams within your organization. Again, product, potentially legal, maybe IT, to understand your data movement patterns and the business requirements for those data movement patterns. Why does your data need to move multiple times within an availability zone? Why does it need to move between regions? Do you need to have data that is copied across multiple availability zones? Do you need that data to be cross-region? These are some examples of really important questions to ask to understand, do you need to continue transferring that data? Because the more you can optimize the way that that data is moving around within AWS, the less money you'll ultimately spend.Pete: Yeah, and this ties into, again as you've noticed, there's a reoccurring theme is th...

Want to give your ears a break and read this as an article? You’re looking for this.Never miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of February 1, 2021 with Corey Quinn.

LinksForrest Brazeal article referenced: https://acloudguru.com/blog/engineering/the-lift-and-shift-shot-clock-cloud-migrationUnconventional Guide: https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/ TranscriptCorey: This episode is sponsored in part by our friends at Fairwinds. Whether you’re new to Kubernetes or have some experience under your belt, and then definitely don’t want to deal with Kubernetes, there are some things you should simply never, ever do in Kubernetes. I would say, “run it at all;” They would argue with me, and that’s okay because we’re going to argue about that. Kendall Miller, president of Fairwinds, was one of the first hires at the company and has spent the last six years the dream of disrupting infrastructure a reality while keeping his finger on the pulse of changing demands in the market, and valuable partnership opportunities. He joins senior site reliability engineer Stevie Caldwell, who supports a growing platform of microservices running on Kubernetes in AWS. I’m joining them as we all discuss what Dev and Ops teams should not do in Kubernetes if they want to get the most out of the leading container orchestrator by volume and complexity. We’re going to speak anecdotally of some Kubernetes failures and how to avoid them, and they’re going to verbally punch me in the face. Sign up now at fairwinds.com/never. That’s fairwinds.com/never.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field.Jesse: I like that. I feel like that's good. That's a solid way to start us off.Pete: Triple F. I am Pete Cheslock.Jesse: I'm Jesse DeRose.Pete: #TripleF. We should get some, I don’t know, jackets made? Mugs?Jesse: Lapel pins? I'm open. I've always wanted a Members Only jacket.Pete: If Guy Fieri can call diners, drive-ins, and dives, “Triple D,” then we can definitely call this Triple F.Jesse: We can definitely make this happen. Pete: It's not my high school transcript, either, we're talking about here. Oh, well, we are back again, continuing our series on The Unconventional Guide to Cost Management with Episode Two: the Cloud is not your data center.Jesse: Yeah, this one's gonna be a fun one. I feel like this is a topic that comes up a lot in conversations, sometimes with clients, sometimes with potential clients that are asking, “What kind of things do you see day-to-day? What are some of the big pain points that you see with your cost optimization work?” And so real quick backstory, make sure that you've listened to the previous few episodes to get some context for this segment that we're doing and get some framing for this Unconventional Guide work that we are discussing. But talking about using the Cloud as a data center, I have a lot of thoughts on this.Pete: Well, hold on a second. Isn't the Cloud just someone else's data center?Jesse: [laugh] I—yeah, you know, this is the same argument of serverless isn't actually serverless. It's just somebody else's computer.Pete: [laugh]. Someone else's Docker container. But really, there's a lot of ways we're going with this one. But we're coming at it from, obviously, a cost management perspective. And the big, bold, unpopular opinion that we're gonna say is, the most expensive way to run an application in the Cloud, is by treating the Cloud as just another data center; it's going to cost you way more than it would cost to run in a normal data center. And this goes to the world of, in the early days of Cloud, people just raging online and in conferences about the Cloud, it's so expensive. And yes, it is so expensive, if you treat it like an antiquated data center.Jesse: And really quick before you get your pitchforks out, there is this concept of ‘lift and shift’ that everybody likes to talk about or ‘technical transformation’ that everybody likes to talk about: moving from a data center into the Cloud, which a lot of people see as this movement where they just uproot everything from their local data center into AWS. And to be clear, we do recommend that. That is a solid strategy to get into the Cloud as fast as possible; just move those workloads over. But it is going to be expensive, and it's not what you ultimately want to stick with long term. So, that's ultimately the big thing to think about here. Yes, lifting and shifting from your data center into the Cloud is absolutely worthwhile. But it creates this shot clock that's now running after your migration is complete, where if you don't move on to all of the services, and opportunities, and solutions that AWS provides that are native solutions, cloud-native solutions, managed solutions, you're going to end up spending a lot more money than you want.Pete: Yeah, “The Lift And Shift Shot Clock” that was a great blog post by Forrest from ACG—ACloudGuru. We'll include a link to that in the [00:04:35 show notes]. It talks about how not only do you have technical debt accruing as you lift and shift, but potentially the brain drain as people get sick of managing this hot mess that you've lifted and shifted over. That doesn't mean you shouldn't do it. You absolutely should get into the Cloud, get into a singular vendor with your workloads as fast as possible so that you can then dedicate resources to refactoring all of that. Don't just forget about it and leave it behind. It's not going to end well for you. And you do have a time; the timer is running. So, when you're only using those core primitives—compute, object store, block store—yeah, you're going to have a pretty fixed cost on your cloud bill. But to Jesse's point, there's a lot of other services. Some of those require an engineering effort. Some of those just involve correctly using an instance type, a storage location that is more specific to its access patterns. I mean, everything is basic as T class instances—for those services that maybe don't use a lot of CPU—to reminding yourself that there are multiple tiers of S3 storage. Even Intelligent Tiering will just tier it for you. So, if you go and store everything on standard S3 storage and use GP2 volumes on EC2, yeah, it's gonna be expensive. And I know that because I look at a lot of Amazon bills, and Jesse does too, and we see the same thing. “Oh, you've got a really high bill.” “Yeah, we spend a lot on EC2.” It's, “Like, oh, let me guess. A lot of, like, I3s and C5s and M5s and a ton of EBS, right?” And they give you all this optionality, and I think it's that choice which is so overwhelming for many folks moving to the Cloud. I mean, that's, that's really the case. It's just, “What do I pick?” There's just so much.Jesse: So, let's talk about ephemerality, especially in the world of compute. Ephemerality really ...

Want to give your ears a break and read this as an article? You’re looking for this link.Never miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of January 25, 2021 with Corey Quinn.

Check out the full unconventional guide here!TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if wanting new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to AWS Morning Brief. I am Pete Cheslock.Jesse: I'm Jesse DeRose.Pete: This is Fridays From the Field. Triple F.Jesse: I feel like we've really got to go full Jean-Ralphio, Parks and Rec there. “Friday From the Feeeeeeeeeeild.”Pete: Yeah, so we're going to need to get an audio cut of that and add some techno beats to it. I think that's going to be our new intro song.Jesse: [imitates techno beats].Pete: Yeah, we're going to take both of those things. I'm glad we got this recorded because that's going to turn into a fantastic song. So, we're back to talk about The Unconventional Guide to Cost Management. And this is the first episode, this is the first of a whole slew of these that we're going to be going through from the field, these different ways that companies can impact their spend. And no, it doesn't mean go and buy the cloud management vendor of the moment to look at your spend or fire up Cost Explorer. Those are all pieces of it, but broader things, the big levers, the small levers, the levers that don't actually go back and forth, but you turn and you would have no idea because it was designed by an Amazon UX engineer.Jesse: Yeah, it's really important to call out that this discussion is looking at your cloud spend from a broader perspective and if you didn't get a chance to listen to our episode from last week, we did a little bit of an intro, framing this entire discussion. Go back and take a listen, if you haven't yet. Really talking about why looking at cloud costs through these different lenses is important. Why are you thinking about cloud cost, not just from the perspective of, “Oh, I'm going to delete these EBS snapshots,” or, “I'm going to tag all my resources,” but why is it important to think about cloud costs from other mediums?Pete: Exactly. So, don't forget, you can go to lastweekinaws.com/QA and put your questions right in that box. Your name is optional. You can just leave your name blank if you don't want anyone to know who you are. Or if you want to say something really nice about me and Jesse, and you just feel a little shy—Jesse: Aww.Pete: —that's fine, too. But just put a question in there. And we're going to dedicate some future episodes to answering those questions and diving a little deeper for those that want to know a little bit more. But as being the first episode, we got to talk about something, so what are we talking about today, Jesse? Jesse: Today we are talking about architecture and architecture context. Now, this is a really, really interesting one for me because the first thing that I think anybody thinks about when they think about cutting costs with their AWS spend is architecture decisions: something related to your infrastructure, whether that's tearing down a bunch of resources, or deleting data that's lying around. But there's a lot more to it than that context is everything. Knowing why your infrastructure is built the way it is, knowing why your application is designed the way it is, is really important to understanding your AWS cloud costs.Pete: This is where I feel like the Cloudabilitys CloudHealth, CloudCheckr Cloud-whatever companies, their products, sadly, fall down. And similar for every Amazon recommendation engine inside of AWS, they all break down. They lack the knowledge and the context of your organization. I remember a really long time ago, I had installed CloudHealth for the first time, and it said, “Hey, we've identified all these servers. They're sitting idle. Do you want us to turn them off for you?” Those servers were actually my very large Elasticsearch cluster. They were idle because if no one's querying them they don't do anything, but they sure do hold a lot of data, and they really do need to be available. So, please, please don't turn those off. But that same thing could happen if you were—you know, due to risk or compliance reasons, you had to run some infrastructure as a warm standby in another availability zone or region. Yeah, sure, it's not taking requests, it’s not doing anything, but that doesn't mean that it's not supposed to be running.Jesse: And this is really getting at one of the first big ideas, which is: work with other teams within the company. Not just other engineering teams, but product teams, possibly also security teams to understand all of the business context for your application and for your infrastructure in terms of data retention, in terms of availability, in terms of durability requirements. Because ultimately, you as a platform engineer, or an SRE, or a DevOps engineer, or whatever the hot new title is going to be a year from now, you need to understand why the infrastructure exists, and you may see servers that are sitting around idly doing nothing, but that's your disaster recovery site that is required by the business, by a service level agreement to be available at a moment's notice if something goes wrong. And so it's really important to understand what those components are and how they work together to build your overall application infrastructure.Pete: Yeah, that's a great point. I mean, having that knowledge that if you've been at a company for years, you've got a lot of this historical knowledge. People have come and gone, they've come, they've done things, they've implemented items, they've brought new features, they've gone. As companies grow may or not— may not be a single person who really truly understand the impact of various changes. I think we saw that most clearly when Amazon had their Kinesis outage: the amount of different services that were impacted was pretty large because it's just all too big for any one person to understand. But that doesn't mean that you shouldn't always continually be working to understand those different usage requirements, and chatting with the non-tech teams. Product teams, I feel like are often ignored in startups because you don't really want more work, and that's what those product teams normally do, right? But they're going to have a lot of context. I remember working in SaaS companies and looking at things like, “This? We don't use this anymore. There's no way we use this. I'm going to turn this off.” And then, I then say, well, the smarter minds prevail. I say, “Well, let me go talk to product people.” And they go, “Oh yeah. We can't get rid of that one super important API because this one client of ours paid us an obscene amou...