AWS Morning Brief

AWS Morning Brief for the week of January 11, 2021 with Corey Quinn.

TranscriptCorey: Software powers the world. LaunchDarkly is a feature management platform that empowers all teams to safely deliver and control software through feature flags. By separating code deployments from feature releases at scale, LaunchDarkly enables you to innovate faster, increase developer happiness, and drive DevOps transformation. To stay competitive, teams must adopt modern software engineering practices. LaunchDarkly enables teams to modernize faster, Intuit, GoPro, IBM, Atlassian, and thousands of other organizations rely on LaunchDarkly to pursue modern development and continuously deliver value. Visit us at launchdarkly.com to learn more.Pete: Hello, and welcome to the AWS Morning Brief. I’m Pete Cheslock.Jesse: I'm Jesse DeRose.Pete: And we're back yet again. We're well into 2021. I mean, about a week or so, right?Jesse: I'm excited. I'm just glad that when midnight struck. I didn't roll back over into January 1st of 2020.Pete: Yeah, luckily, it's not a Y2K scenario. I don't think we have to deal with the whole date issues until, what, 2032 I think, whatever that the next big Y2K-ish date issue is going to be. I'm hopefully retired by the time that that happens. Jesse: That's future us problem. Pete: Yeah. Future us problem, absolutely. Well, we've made it. We've made it to 2021, which is a statement no one thought they were going to say last year at this point.Jesse: [laugh].Pete: But here we are. And today, we're talking about an interesting topic that may bring us some hate mail. I don't know. You tell me, folks that are listening. But we're seeing this more and more in our capacity as cloud economists working with clients here at The Duckbill Group, that folks who are running Kubernetes—whether it's EKS, or they're running it on EC2 using maybe, like, an OpenShift—are actually spending more than people who are using other primitives within AWS. So, we wanted to chat a little bit about why we think that is, and some of the challenges that we're seeing out there. And we would love to hear from you on this one. If you are using Kubernetes in any of the ways that we're going to talk about, you can actually send us a story about how you're doing that and maybe answer some of these questions we have, or explain how you're using it. If you go to lastweekinaws.com/QA to ask us questions—not quality assurance—but go to QA for asking us questions. You can put in your information, you can add your name, it's optional if you want. You can be completely anonymous and just tell us how much you enjoy our wonderful tones and talking about technology. So, Kubernetes. Why is this the thing, Jessie?Jesse: I feel like when it first came out, it was the hot thing. Like, everybody wanted Kubernetes, everybody wanted to be Kubernetes, there were classes on Kubernetes, there were books on—like, I feel like that's still happening. I think it has amazing potential in a lot of ways, but I also feel like… in the same way that you might read the Google SRE book and then immediately turn to your startup team of three people and say, “We're going to do everything the way that Google does it,” this isn't always the right option.Pete: Feel like the Google SRE book is, like, The Mythical Man Month, which is, the book that everyone wants to quote, the name of the book, but none of those people have ever actually read the book.Jesse: Yeah, there's lots of really great ideas, but just because they're great ideas that worked well for a large company at scale doesn't necessarily mean that they're going to be the same right ideas for your company.Pete: And also, we're both fairly grizzled former system administrators and operators; Kubernetes is not the first, kind of, swing of the bat at this problem. I mean, we've had Mesos which, it's still around but not as hip and cool; we've had OpenStack. Does—remember when all the Kubernetes people were all like, “Nope, OpenStack is going to be the greatest thing ever.” So, needless to say, we are a little jaded on the topic.Jesse: You can't forget about Nomad, either, from HashiCorp built cleanly into HashiCorp’s Hashi stack with all of their other amazing development and deployment tools. Pete: Yeah. I mean, this is a problem that people want to solve. But in the rise of Cloud, on Amazon I always struggled with why it was needed. And we're going to talk a little bit about that. So, again, what is Kubernetes? I hope people are listening that would know this, but maybe not. It's an abstraction layer for scheduling workloads. It's the solution to the Docker problem. Like, a container is great. I have a container, it is a totally self-contained application, ready to go, my configuration, my dependencies. And now I need a place to run it. Well, where do I run this container? Well, pre-Kubernetes, Jessie, you'd probably use something like ECS—the Elastic Container Service—might be a way that you could schedule some workloads. Jesse: Or maybe if you just wanted to run a single virtual machine somewhere and run that container in the virtual machine, you might do that as well. Pete: Yeah, that was how a lot of the earliest users of Docker were just running Docker: they were just running the containers as applications—because that's what they are—on their bare EC2. They would just run some EC2 and run a Docker container on there. And there were benefits to that. You got this isolated package deployed out there not having to worry about dependencies. You have to worry about having the right Python dependencies or Ruby dependencies. It came with everything it needed, and that was a big solution. Now Kubernetes, I think, brings this really interesting concept that I like. It's this API that theoretically you could use in a lot of different places. If you now have this API to deploy your application anywhere there's a Kubernetes cluster, does this solve vendor-lock-in? Could you use Kubernetes to solve some of these issues that we see?Jesse: You could use Kubernetes to solve vendor-lock-in in the same way that you could use multi-cloud to solve vendor lock-in. Again, it is a solution to the problem, but is it the right solution for your company?Pete: That is always the question I feel like I would ask folks when they were using Kubernetes is, I would always ask why they were using it. I honestly will say I never got—I don’t want to say wouldn't say never; that's not fair. I rarely would get a good answer. It was often like a little bit of operational FOMO—you know, the fear of missing out on the next hottest thing, which of course, that's never a good way to pick your architecture stack. Now, that being said, at a previous company, we were investigating Kubernetes to solve a problem with our stateless applications—because I in no way trusted it to run anything stateful. None of my databases I wanted on it. But it is a great way to put more control into my developers’ hands-on deploying their applications. We ran predominantly C class instances on EC2. And th...

Want to give your ears a break and read this as an article? You’re looking for this link.SponsorsVeeamExtraHopNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of January 4, 2021 with Corey Quinn.

Links#AWSWishList@AWSWishList AccountFollow Pete + Jesse on TwitterTranscriptCorey: When you think about feature flags (and you should), you should also be thinking of LaunchDarkly. LaunchDarkly is a feature management platform that lets all your teams safely deliver and control software through feature flags by separating code deployments from feature releases at massive scale (and small-scale too), LaunchDarkly enables you to innovate faster, increase developer, happiness (which is more important than you think), and drive transformation throughout your organization. LaunchDarkly enables teams to modernize faster. Awesome companies have used them, large, small, and everything in between. Take a look at launchdarkly.com to learn more and tell them that I sent you. My thanks again for their sponsorship of this episode.Pete: Hello and welcome to the AWS Morning Brief. I am Pete Cheslock.Jesse: I'm Jesse DeRose.Pete: We are welcomed yet again with Amy Negrette.Amy: Hello.Pete: We are here. We made it. It is actually 2021.Jesse: I can tell you flying cars: definitely a thing. World peace: we're close, we're so close.Pete: We're so close. Well, guess what? We made it, we survived 2020. And with it, we brought with us part two of the #awswishlist. So, this is where we went through—especially as leading up to re:Invent and getting through re:Invent—we went through and looked at the Twitter hashtag of #awswishlist so that we could pick out some of our favorite things, some #awswishlist items that we think are important to us, or just interesting in their own right. We'll include the link to these tweets in the [00:01:57 show notes]. So definitely go check that out, and you can check out the conversation, or maybe follow some of that to see when things actually come around. But yeah, we'll just walk through some of the things we found that were pretty interesting and chat about why we hope Amazon includes them into a future release. So, one thing that I saw which I thought was pretty interesting because I run into this problem also, is a way of downloading data from various third party locations directly into S3, Dynamo, or some sort of data store location. Essentially, it'd be awesome to just completely get rid of having services around, or Fargates, or Lambdas set up for downloading data from places that—how cool would it be? And this is, again, not an enterprise-y type feature, but just, like, a personal thing of how cool would it be to be, like, I want to take this ISO from a place and just put a URL in S3 and say, “Put that thing in this thing,” and call it a day. So, again, a personal complaint of mine plus, also, someone else tweeted it, so there's two people out there that want this—at least—so therefore Amazon, you got to build it for me.Amy: Those are the rules.Pete: Those are the rules. Right. Right, Amy, those are the rules. Jesse: And I feel like, let's be honest, that ISO that you want to download anyway is probably living in S3 somewhere else anyhow. So, it's just moving bucket to bucket.Pete: Someone has that, you know, Slackware ISO that I've been looking for, from, you know, 2001. It's in someone else's bucket; just let me have it myself. Exactly. Amy, what did you find in your discovery of the #awswishlist hashtag?Amy: This is a thing that I think really should be on any of these on-demand pay-as-you-go services because AWS really targets those [00:03:48 unintelligible] markets for a lot of their serverless deployments. And this actually came from one of my friends who had this problem on Twitter, where you need to be able to set a maximum on on-demand spend, let's say in his case, Dynamo. So, you don't hypothetically build in a loop and spend a whole bunch of money. Pete: Yeah.Amy: And really, it should be in anything that does that. If it's not telling you something where I'm only wanting to run this much because it's on-demand, then you should be able to control that spend somehow.Pete: And with the—what is it—millisecond billion on Lambda, you can get really granular bills for your poorly architected Lambda functions. Jesse: I feel like computers are the best because they'll do exactly what you want them to do, except for when they do what you tell them to do and not what you actually want them to do, and that drives me absolutely insane. So, I'm with you. I think that this is a great opportunity.Amy: That problem will be solved when the robots take over.Pete: [laugh]. One of my favorite discoveries of doing our kind of Duckbill cost optimizations where we dive into people's spend and help them architect things new was finding a Lambda function that was taking longer and longer to execute—meaning, costing more money—by putting more and more data into a poorly configured Dynamo table that was also causing it to take longer and longer. And so not only did you have a Dynamo table that was poorly configured, taking this data and taking longer to do it, you were just getting a hit on both sides. It happens.Jesse: That hurts my soul. Pete: So, what’d you find, Jesse? What was some of the good wishlist items that you're hoping for in 2021?Jesse: So, I come from a background of a lot of infrastructure as code I've worked a lot with Terraform, I know enough about Chef to be dangerous to your production environment. One thing that I saw a couple people tweet about that I would love to see is mock AWS API endpoints for, effectively, unit tests for a lot of infrastructure as code. Because if you think about when you're building infrastructure as code, the only way that you can really test it is by running it, by actually seeing, “Can I actually create the resources that I think I'm creating with this infrastructure as code content?” So, I would love to see maybe a feature flag for AWS services through the API where you can say, “Hey, don't actually create this RDS database or this EC2 instance, but just return the results as if I did create it. Maybe leave the Instance ID blank or something like that.” And then you, in writing your unit tests, can confirm all the details that you would expect to see in that response. Pete: I feel like there was a—Atlassian, maybe, had a project that was something like this, some sort of a way of unit testing these things. Again, it was something on GitHub, so even if it was associated with a large publicly traded enterprise, I'm sure it's fallen into disrepair at this stage.Jesse: [laugh]. I will say I found an open-source tool looking into this one, called LocalStack that allows you to basically spin up an instance on your local machine that acts as the AWS API endpoint so that it actually creates this mock endpoint for you locally on your machine. But effectively, I'd love to see th...

Want to give your ears a break and read this as an article? You’re looking for this link.SponsorsExtraHopLinodeNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of December 28, 2020 with Corey Quinn.

Links#AWSWishList@AWSWishList AccountFollow Pete + Jesse on TwitterTranscriptCorey: This episode is sponsored in part by our friends at Linode. You might be familiar with Linode; they’ve been around for almost 20 years. They offer Cloud in a way that makes sense rather than a way that is actively ridiculous by trying to throw everything at a wall and see what sticks. Their pricing winds up being a lot more transparent—not to mention lower—their performance kicks the crap out of most other things in this space, and—my personal favorite—whenever you call them for support, you’ll get a human who’s empowered to fix whatever it is that’s giving you trouble. Visit linode.com/screaminginthecloud to learn more, and get $100 in credit to kick the tires. That’s linode.com/screaminginthecloud.Pete: Hello and welcome to AWS Morning Brief. I am Pete Cheslock. I'm joined yet again with Jesse DeRose. We are also excited to re-invite recurring guest for number two, Amy Negrette. Say hello, Amy.Amy: Hello.Pete: So, we are here. This is Christmas. Or should I say Christmahanukwanza. Jesse: So, close. That works. Pete: So, close. But it's the Christmahanukwanza episode—Hanu—hanukwanza—Jesse: Christmashanukwanzika. Pete: And if you thought Hanukkah was spelled a bunch of different ways, Christmahanukwanza is spelled a lot of different ways. And we are here to talk about the #amazonwishlist, which is honestly one of my favorite hashtags to follow on Twitter—#awswishlist. It is pretty popular, it's heavily used.Jesse: It was actually so heavily used that they made a specific @awswishlist account, basically, specifically to follow a lot of these hashtags, and to re-highlight a lot of these hashtags, especially when some of the wishes are actually fulfilled. Pete: Yeah, I think it's a great thing, and if I was an Amazon product manager, I would love this too because just talk about making my job a lot easier, I guess. Jesse: One thing that I do want to call out, I was looking through a number of the tweets going around for the hashtag#awswishlist, and I noticed that there was some of the responses from AWS folks, which one I'd love to say thank you, AWS for actually taking this seriously and actually responding to folks in conversation on Twitter for these wishlist items. There was one that I found where the person directed the original poster to an AWS support page, which was basically AWS’s, like, ‘Contact Us’ page. And the Contact Us page basically said, “Hey, if you have some questions, here's what you should do. I have some questions that could help improve an AWS product or service, how can I send feedback to AWS?” And all the answers were, “Click the feedback button on the page that you're on, either in the AWS console or the AWS documentation, or contact AWS support directly.” So, close—Pete: Did you just tell me to go F myself there, Jesse? [laugh].Jesse: [laugh]. I didn't maybe say it in so many words, but I think I did.Amy: I absolutely love it when a support page says, “Maybe you should just do it yourself.” And I'm like, “Well if I did, I probably wouldn't have been here in the first place.”Pete: Exactly. So, what we decided to do, what we thought would be kind of fun, is to troll through the Twitter #awswishlist hashtag and take a look at what people were saying, especially because it's a lot busier around the pre to current re:Invent time. And so independently each of us put together a list of things that—I mean, at least I could speak for myself—I thought were interesting, or things that I thought would be cool to have. And yeah, we're just going to talk about them and see from there. So, we'll include a link to each of these tweets in the [00:04:18 show notes] so you can check them out, and also so you can see the conversation on them. What was also cool, I just want to call out is that some of these that we saw on there, at least that I saw have been resolved by re:Invent time. One was AWS CloudShell that was announced recently at re:Invent, someone was saying I want is this AWS CloudShell thing because other vendors have this: Azure has this, Google has this. So, here's a scenario where Amazon was catching up. So, I thought that was pretty cool to see. So, I'm going to kick it off because, whatever, I'm here, and I got my list in front of me. So, this is actually related to the CloudShell one, which I thought was interesting. So, there was some conversation online about CloudShell, and this is maybe potentially allowing people to remove the need of having a bastion host, which, how cool is that you don't have to run those anymore? Jesse: Oh, yeah. Pete: And so there was a question around, “Well, does my identity get a home directory?” Which sounds like the answer was “Yes.” But the question mark there had to do when using AWS SSO because it has to do with the IAM principle, it's what comes back from the sts get-caller-identity. So, if you are using one of the different Federation technologies, your actual identity could be different for each one. And so that's a wishlist item that I could definitely be on board with because if you're dealing with IAM roles or Federation, and your home directory is never the same, that can be kind of annoying. Jesse: I cannot tell you how many times I have downloaded a file or put a file somewhere on a bastion host, gone away to a different project, come back to it, or SSH’ed into the same bastion host and wondered why it wasn't there anymore, only to realize that I was on a different bastion host in a different environment, or that the data had been purged every so often for security or cleaning purposes. I would absolutely love clean roles and just really, really well defined boundaries on this. Coming from somebody who uses different AWS accounts on a regular basis for the different clients that we work with, I would just love to see this really kind of clean structure of AWS, IAM usage, and user management and security.Pete: And, Jesse, we saw similar issues, I believe, when we were playing around with QuickSight, and Federation, and IAM so—Jesse: Oh, yes.Pete: Hopefully that gets a little bit fixed up. But anyway, I thought that was a pretty interesting one. Amy, what did you find in your discovery of the Amazon wishlist hashtag?Amy: I did find one for X-Ray support in API Gateway HTTP API. Again, one of the worst, longest names of any service, and EventBridge, which surprisingly, one that this hasn't happened yet, but two, [00:07:12 unintelligible] for me is kind of a double-edged sword where it's one of those services that everyone needs, but als...

Want to give your ears a break and read this as an article? You’re looking for this link.SponsorsExtraHopLinodeNever miss an episodeJoin the Last Week in AWS newsletterSubscribe wherever you get your podcastsHelp the showLeave a reviewShare your feedbackSubscribe wherever you get your podcastsWhat's Corey up to?Follow Corey on Twitter (@quinnypig)See our recent work at the Duckbill GroupApply to work with Corey and the Duckbill Group to help lower your AWS bill

AWS Morning Brief for the week of December 21, 2020 with Corey Quinn.