AWS Morning Brief

Links:The cloud economist starter kit: https://www.lastweekinaws.com/podcast/aws-morning-brief/cloud-cost-management-starter-kit-2/TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.Amy: I’m Amy Negrette.Jesse: This is the podcast within the podcast where we like to talk about all the ways we’ve seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. Today, we’re going to be talking about balancing cost optimization work against feature work.Amy: Buckle up everyone. I’ve got a lot of thoughts about this. Just kidding. It’s just the one: don’t.Jesse: You heard it here first, folks. Don’t. Amy Negrette just says, “Don’t.”Amy: Don’t. [laugh].Jesse: So Amy, does that mean, don’t balance the work?Amy: More like don’t choose. It’s always hard to make the argument to take an engineer off of feature work. This goes for all sorts of support tasks like updates and documentation, and as a group, we figured out that trying to put those off until an engineer has time to do it is not going to be a thing that becomes prioritized, it eventually gets deprioritized, and no one looks at it. And that’s why DocOps is the thing. It’s a process that now gets handled as part of and in parallel with software development.Jesse: Yeah, I’ve had so many conversations in previous companies that I’ve worked for, where they basically said, “Well, we don’t have time to write documentation.” Or they will say, “The code is the documentation.” And, to their credit, there are a lot of places where the code is very cleanly documented, but if somebody is coming into this information for the first time and they don’t have technical knowledge or they don’t have deep expertise in what you’re looking at, they need documentation that is clear, understandable, and approachable. And it is so difficult to find that balance to actually make sure that that work is part of everything that you do.Amy: And I think what the industry has decided is that if you make it a requirement for pull requests that if you’re going to make a change, you have to document that change somewhere, and that change if it has any kind of user impact, it will be displayed alongside it. That’s the only way to make it a priority with software. And cost optimization has to be treated in a similar respect.Jesse: Yeah, so let’s talk about cost optimization as a process. To start, let’s talk about when to do it. Is this something that we do a little bit all the time, or do we do it after everything’s already done?Amy: I know I just cited CostOps as a good model for this, even though that’s literally what we cannot do. We can’t treat cost optimization as something we do a little bit along the way because, again, speaking as an engineer, if I’m allowed to over-optimize or over-engineer something, I’m going to take that opportunity to do that.Jesse: Absolutely.Amy: And if we’re going to do project-wide cost optimization, we need to know what usage patterns are, we need to have a full user and business context on how any system is used. So, if we do a little at each step, you get stuck in that micro-optimization cycle and you’re never actually going to understand what the impact of those optimizations were. Or if you spent too much time on one part over-optimizing another part.Jesse: It’s also really hard if this is a brand new workload that you’ve never run in the cloud before. You don’t necessarily know what the usage is going to be for this workload. Maybe you have an idea of usage patterns based on some modeling that you’ve done or based on other workloads that you’re running, but as a whole, if this is a brand new workload, you may be surprised when you deploy it and find out that it is using twice the amount of resources that you expected, or half the amount of resources that you expected, or that it is using resources and cycles that you didn’t expect.Amy: Yeah. We’ve all been in the situation, or at least if you work with—especially with consumer software—that, you’re going to run into a situation where the bunch of users are going to do things that you don’t expect to happen within your application, causing the traffic patterns that you predicted to move against the model. To put it kindly. [laugh].Jesse: Yeah. So, generally speaking, what we’ve seen work the best is making time for cost optimization work maybe a cycle every quarter, to do some analysis work: to look at your dashboards, look at whatever tooling you’re using, whatever metrics you’re collecting, to see what kind of cost optimization opportunities are available to you and to your teams.Amy: So, that comes down to who’s actually doing this work. Are we going to assign a dedicated engineer to it in order to ensure it gets done? Anyone with the free cycles to do it?Jesse: See, this is the one that I always love and hate because it’s that idea of if it’s everyone’s responsibility, it’s no one’s responsibility. And I really want everybody to be part of the conversation when it comes to cost optimization and cloud cost management work, but in truth, that’s not the reality; that’s not the way to get this work started. Never depend on free cycles because if you’re just waiting for somebody to have a free cycle, they’re never going to do any work. They’re never going to prioritize cost optimization work until it becomes a big problem because that work is just going to be deprioritized constantly. There’s a number of companies that I worked for in the past who did hackathons, maybe once a quarter or once every year, and those hackathons were super, super fun for a lot of teams, but there was a couple individuals who always picked up feature work as part of the hackathon, thinking, “Oh, well, I didn’t get a chance to work on this because my cycles were focused on something else, so now I’ll get a chance to do this.” No, that’s not what a hackathon is about.Amy: You don’t hack on your own task list. That’s not how anything works.Jesse: Exactly. So instead, rather than just relying on somebody to have a free cycle, kind of putting it out there and waiting for somebody to pick up this work, there should be a senior engineer or architect with knowledge of how the system works, to periodically dedicate a sprint to do this analysis work. And when we say knowing how the system works, we’re really talking about that business context that we’ve talked about many, many times before. A...

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/turn-that-sh—-offNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of May 31, 2021, with Corey Quinn.

Links:Ted Talk: The Science of Productive ConflictTranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.Amy: I’m Amy Negrette.Jesse: This is the podcast within a podcast where we talk about all the ways we’ve seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. Today, we’re going to talk about the people side of technical projects, especially people who might introduce roadblocks for completing technical projects. Now, you might be thinking to yourself, “Jesse, Amy, that sounds like it is not about AWS.” But let me assure you that any project involving AWS is going to involve multiple different personalities approaching the project from different angles, who ultimately all have the same solution in mind, but have different ideas about how to get that problem done, or different ideas about what’s the right thing that ultimately get done to begin with. So today, we want to talk about that: we want to dive into how can you have really rewarding conversations with those folks? How can you better engage people who are intentionally or unintentionally difficult?Amy: We want to be very clear that we are not trying to come after anyone. Every time that I’ve gotten an engagement, it isn’t because someone means to be difficult, but maybe there’s a project timeline, maybe there’s something else getting in the way of them being able to fully be present for that specific part of the engagement, and maybe that’s just what’s causing friction, causing speed bumps. And we’re all well aware of this. Jobs are hard, and especially this sort of work can be difficult. So, first of all, we totally understand, and this is just more about how to get everyone moving in the same direction at the same pace.Jesse: Yeah, absolutely. I mean, especially with the pandemic going on right now, everybody’s doing remote work, some people have never actually met their teammates in person and they’re expected to work together efficiently, and quickly, and easily. It’s hard.Amy: It also doesn’t help that when we do come in, we come in under the context of a cost optimization project, or some other efficiency-type title. And that sounds a little like the Bobs from Office Space, which I bring up a lot, especially during internal meetings. And it makes it sound like we’re going to come in to shake a bunch of things up and look for inefficiencies where there aren’t any, which is truly not the case. And it can cause a lot of insecurity, especially about how someone thinks that they’re doing their job, or that their job is somehow going to be impacted by what our suggestions are. It may not just be us, but it may be another migration consultation suite, where someone’s coming in to change the architecture that they’ve worked on for a long time, and that can put a lot of people in a state of unease.Jesse: And I think it’s also important to note that it’s not just about an external party coming in like Duckbill Group or another external, third-party consulting service, or technical group. It could be an internal separate team. It could be your internal cloud cost management team that is starting conversations with development teams saying, “Hey, I want to better understand how you’re using AWS. I want to understand some of these cost optimization opportunities.” Even in situations like that where all of these conversations are internal within the company, even within teams, there are still multiple different personalities, multiple different people approaching the problem from different angles, and it’s still really, really important to make sure that you approach them collaboratively.Amy: And ultimately, we wanted to be clear that what we’re going to be talking about is helping people think differently into a growth mindset, and being able to do this work without anyone feeling shame or embarrassment.Jesse: Yeah. Growth mindset is so critical. It’s something that I love to talk about ad nauseum, and so I won’t dive into it too deeply here, but—Amy: That’s another episode.Jesse: [Laugh]. Exactly. Growth mindset is so important for folks in technology teams, especially in today’s technology era where there’s just so much constantly innovating. There’s so much new constantly going on around you, to new technologies, new teams, new ideas, new ways of doing things, new processes, new tools; it’s really important to be open-minded to learning those different things. You don’t have to use every single one of them, but be open-minded to different people approaching problems from different perspectives and different angles.Amy: Having to face all of this uncertainty will cause some to not be the most cooperative when they have to start reacting into these situations, whether it is an internal change that’s happening, or if it’s an external consulting group; they can start coming back and taking a various sort of stance, and just like being back in middle school, sometimes standing up to a bully is simply how you have to succeed because it’s not about dominating, it’s about compromise and trying to find out what you’re trying to do and find that common ground.Jesse: Yeah, absolutely. I think that’s the most important part here because when we talk about working with other personalities that are different than yours and having conflict, it’s not about dealing with them; it’s not about overcoming them from the perspective of winning the argument, so to speak. It’s about how do you compromise? How do you effectively find that common ground and move forward together? And sometimes it’s just about sharing context, it’s just about sharing that mental model that you have that might be different than the mental model that this other person has, or maybe the other team has.Like for example, some teams that we’ve talked to can’t make a cost optimization change due to security, or legal, or product SLA restrictions, but maybe the person who’s coming in from the cloud cost management team or cloud cost management side doesn’t know that because they aren’t as familiar with the product.Amy: But it can also just be a staffing issue. These projects take work, and if an engineering team is already stressed and stretched to the edge, they’re not going to have the resources, and they don’t want to be the ones to say we simply don’t have the manpower to do this.Jesse: Yeah, absolutely. And it’s so, so important to be able to identify those bottlenecks or identify those constraints. Because ultimately, if you give a team that already has a ton of things on their roadmap new work and say, “Hey, cost optimizati...

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/the-17-ways-to-run-containers-on-awsNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of May 24, 2021 with Corey Quinn.

Transcript Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.Amy: I’m Amy Negrette.Jesse: This is the podcast within a podcast where we talk about all the ways we’ve seen AWS used and abused in the wild. With a healthy dose of complaining about AWS for good measure. Today, we’re going to be talking about build versus buy. I feel like this is really kind of a classic engineering conversation. Amy, what is the build versus buy idea?Amy: It’s really the idea of whether you decide to use a managed service or SaaS product versus rolling your own and building yourself. It’s very easy to do these days with a few watches on YouTube, maybe some blog articles. You can also do repairs on my house, which is why I always have to get repairs done on my house. [laugh].Jesse: [laugh]. Yeah, I feel like as much as I love the world of HGTV and the DIY network, I think I can do more than I actually can and I feel like it’s probably a lot safer to just let a professional take the reins. I mean, there’s so many certification programs that teach you how to build and manage your own engineering things, your own distributed databases, your own Kubernetes clusters, your own streaming data platform, and it’s really great to understand the fundamental building blocks of these systems, it’s really great to understand how they work so that ultimately if you are consuming from them or managing them, that you understand the ins and the outs of the system. But the question becomes, do you really need to be the one that’s managing that system? Do you really need to be the one spending your time managing that system on top of writing code for your microservices, on top of managing the architecture, the application, all of the components of your service that are critical?Amy: So, I guess what we really want to decide is, in what use cases is it okay to build something from scratch, and when is it okay to, essentially, just go to the market and look for something that’s made already?Jesse: Yeah. And I think that’s the main question that a lot of folks ask: what is the defining line? What are the questions they should think about as they are choosing to build versus buy?Amy: I think if you want to really look at building a product, and really from the ground up—you have this product in mind and you want to do all the architecture, control it end-to-end—unless this is your core product feature or you’re going to package it for either internal or public release, you almost always—you don’t want to build this yourself because someone has probably built it in a way that’s not going to cause your engineers time or money. Unless it is going to directly make you money, then yes. If this is tied to your product income and your product revenue, please build it yourself. It avoids a lot of licensing issues, you do get to control how it works, how you want it to work. But that said a lot of products, just a bunch of assassins in a trench coat anyway, so—Jesse: [laugh].Amy: —it really depends on what’s important to you.Jesse: Yeah, I feel like this is one of the biggest pitfalls that I see in a lot of organizations where they think about how they want to build out an architecture and they choose that a solution like, stateful distributed service is going to be the right thing that they want. And one of the developers says, “Oh, that’s easy. I can build that in a weekend.” And then they go off and build it, and then they’re stuck managing that system for all of eternity when that’s not the primary purpose of the team that they’re working on, that’s not the primary purpose of the product that they’re working on. So, if you’re going to build something that is directly related to your product, directly related to your business use case, directly related to how your company is making money, something that is absolutely your bread and butter, you definitely want to build that rather than buying that off the shelf.Because building it will give you that great opportunity to focus on controlling all the ins and outs of the system, understanding all the parts of the system, finding the flexibility when you need flexibility, really fine-tuning and honing all the parts of the system in the way that you need it to work so that ultimately your organization is getting the best bang for their buck out of the system, whereas in a lot of cases, you’re not going to get the same level of flexibility from an off the shelf solution.Amy: And especially if you’re going to go in and planning to build your own supporting product, make sure—and I said this before, I’ll say it again—you do check the licenses of any libraries and any SaaS products you use to build it because I reinvented the wheel plenty of times in my career, specifically because I worked in a place where the licensing we were allowed to use would not allow us to use very specific products.Jesse: Yeah. That’s such a critical business risk and something that I think not every engineer is fully aware of. And to be clear, I don’t think that’s the engineer’s fault. I think that’s part of best practices that every organization can get better at to make sure that everybody understands, what are our limitations on using modules, using open-source solutions from the internet? How can we make sure that we ultimately aren’t creating additional unnecessary business risk?Amy: When do we go shopping?Jesse: [laugh]. Yeah, let’s go shopping. Let’s say you’ve decided that the piece of software that you want is not part of your bread and butter, like we were saying. If it’s not part of your organization’s primary product, primary use case, don’t waste engineering time building it for yourself, pay a vendor or a subject matter expert to build it for you—or to manage it for you, even—and then call it a day. It is absolutely worth those trade-offs. The additional cost of paying somebody else to manage it for you is absolutely worthwhile because you then get the opportunity to stay focused on the things that are most important to your team and your business.Corey: If your mean time to WTF for a security alert is more than a minute, it’s time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you’re building a secure business on AWS with compliance requirements, you don’t really have time to choose between antivirus or firewall companies to help you secure your stack. That’s why Lacework is built from the ground up for the cloud: low effort, high visibility, and detection. To learn more, visit lac...

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/New-CEO-Onboarding-at-AWSNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of May 17, 2021 with Corey Quinn.

TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.Amy: I’m Amy Negrette.Jesse: This is the podcast within a podcast where we talk about all the ways we’ve seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure because I mean, who doesn’t love to complain about AWS? I feel like that’s always a good thing that we can talk about, no matter the topic. Today, we’re going to be talking about the ‘cloud cost management starter kit.’ So, the starter kit seems to be a big fad that’s going around. If you’re listening to this episode, you’re probably thinking, “It’s already done. It’s over.”But I still want to talk about it. I think that this is a really relevant topic because I think a lot of companies are trying to get started, get their hands started in cloud cost management. So, I think this would be a great thing for us to talk about: what’s in our cloud cost management starter kit?Amy: And it really will help answer that question that I get asked a lot on: what is even a cloud economist, and what do you do?Jesse: Yeah, I mean, given the current timeframe, I haven’t gone to any parties recently to talk about what I do, but I do feel like anytime I try to explain to somebody what I do, there’s always that moment of, “Okay. Yes, I work with computers, and we’ll just leave it at that.”Amy: It’s easier to just think about it as we look at receipts, and we kind of figure things out. But when you try to get into the nuts and bolts of it, it’s a very esoteric idea that we’re trying to explain. And no, I don’t know why this is a real job. And yet it is.Jesse: This is one of the things that always fascinates me. I absolutely love the work that I do, and I definitely think that it is important work that needs to be done for any organization, to work on their cloud cost management best practices, but it also boggles my mind that AWS, Azure, GCP, haven’t figured out how to bake this in more clearly and easily to all of their workflows and all their services. It still boggles my mind that this is something that exists as—Amy: As a thing we have to do.Jesse: As a thing we have to do. Yeah, absolutely.Amy: Well, the good news is, they’re going to change their practices once every six weeks, and we’ll have a new thing to figure it out. [laugh].Jesse: [laugh]. So, let’s get started with the first item on our cloud cost management starter kit. This one is something that Amy is definitely passionate about; I am definitely passionate about, as well. Amy, what is it?Amy: Turn on your CUR. Turn on your CUr. If you don’t know what it is, just Google AWS CUR. Turn it on. It will save you a headache, and it will save anyone you bring in to help you [laugh] [unintelligible 00:02:59] a huge headache. And it keeps us from having to yell at people, even though that’s the thing that if you pay us to do it, we will totally do it for you.Jesse: If you take nothing away from this episode, go check out the AWS Cost and Usage Report—otherwise known as CUR—turn it on for your accounts, ideally enable it in Parquet format because that’s going to allow you to get all that sweet, sweet data in an optimized manner, living in your S3 bucket. It is a godsend. It gives you all the data from Cost Explorer, and then some. It allows you to do all sorts of really interesting business intelligence analytics on your billing data. It’s absolutely fantastic.Amy: It’s like getting all of those juicy infrastructure metrics, except getting that with a dollar sign attached to it so you know what you actually doing with that money.Jesse: Yeah, this definitely is, like, the first step towards doing any kind of showback models, or chargeback models, or even unit economics to figuring out where your spend is going. The Cost and Usage Report is going to be a huge first step in that direction.Amy: Now, the reason why we yell at people about this—or at least I do—is because AWS will only show you the data from the time that it is turned on. They do have it for historical periods, but if you enable it at a specific point, all of your reports are going to start there. So, if you’re looking to do forecasting, or you want to be able to know what your usage is going to be looking like from this point on, turn it on as early as possible.Jesse: Absolutely. If you are listening to this now and you don’t have the CUR enabled, definitely go pause this episode, enable it now, and come back and listen to the rest of the episode because the sooner you have the CUR enabled, the sooner you’ll be able to get those sweet, sweet metrics for all of your—Amy: And it’s free.Jesse: [laugh]. Yeah, that’s even the more important part. It’s free. There’s going to be a little bit of data storage costs if you send this data to S3, but overall, the amount of money that you spend on that storage is going to be optimized because you’re saving that CUR data in Parquet format. It’s absolutely worthwhile.All right, so number two; the second item on our cloud cost management starter kit, is getting to know your AWS account manager and account team. This one, I feel like a lot of people don’t actually know that they have an AWS account manager. But let me tell you now: if you have an AWS account, you have an AWS account manager. Even if they haven’t reached out to you before they do exist, you have access to them, and you should absolutely start building a rapport with them.Amy: Anytime you are paying for a support plan, you also have an account manager. This isn’t just true for AWS; I would be very surprised for any service that charged you for support but did not give you an account manager.Jesse: So, for those of you who aren’t familiar with your account manager, they are generally somebody who will be able to help you navigate some of the more complex parts of AWS, especially when you have any kind of questions about your bill or about technical things using AWS. They will help you navigate those resources and make sure that your questions are getting to the teams that can actually answer them, and then make sure that those questions are actually getting answered. They are the best champion for you within AWS.If you have more than a certain threshold of spend on AWS, if you’re paying for enterprise support, you likely also have a dedicated technical account manager as well, who will be basically your point person for any technical questions. They are a great resource for any technical questions, making sure that your technical questions are answered, making sure that any conce...