Turn on AWS Cost Anomaly Detection Right Now—It’s Free (Whiteboard Confessional)

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Transcript

Corey: This episode is sponsored in part by Catchpoint. Look, 80 percent of performance and availability issues don’t occur within your application code in your data center itself. It occurs well outside those boundaries, so it’s difficult to understand what’s actually happening. What Catchpoint does is makes it easier for enterprises to detect, identify, and of course, validate how reachable their application is, and of course, how happy their users are. It helps you get visibility into reachability, availability, performance, reliability, and of course, absorbency, because we’ll throw that one in, too. And it’s used by a bunch of interesting companies you may have heard of, like, you know, Google, Verizon, Oracle—but don’t hold that against them—and many more. To learn more, visit www.catchpoint.com, and tell them Corey sent you; wait for the wince.

Pete: Hello and welcome to the AWS Morning Brief: Whiteboard Confessional. Corey is still not back. Of course, he did just leave for paternity leave, so we will see him in a few weeks. So, you're stuck with me, Pete Cheslock, until then. But luckily, I am joined again by Jesse DeRose. Jesse, thanks again for joining me today.

Jesse: Thank you for having me. You know, I have to say I love recording from home. I can't see the look in our listeners’ eyes as they glaze over while we're talking. It's absolutely fantastic.

Pete: It's fantastic. It's like a conference talk, but there's no questions at the end. It's the best thing ever.

Jesse: Yeah, absolutely. I love it.

Pete: All right. Well, we had so much fun last week talking about a new service. Although it turns out it was new to us. It was the AWS Detective—or Amazon Detective. There's still some debate about what the actual official name of that service is. For some reason, I thought that service came out in the summertime, but it turns out it was earlier in the year. So, still a great service, AWS Detective—or Amazon Detective, whichever way you go with that one—but we had such a fun time talking about a new service that we had the opportunity of testing out an actual brand new service. This was a service that was just announced last Friday. And that's the AWS Cost Anomaly Detection service. Jessie, what is this service all about?

Jesse: So, you likely would notice if your AWS spend spiked suddenly, but only the really, really mature organizations would be able to tell immediately which service spiked. Like, if it's one of your top five AWS Services by spend, you'd probably be able to know that it's spiked, you'd probably be able to see that easily in either your billing statement or in Cost Explorer. But what if you're talking about a spike in a much smaller amount of spend, that's still important to you, but it's a service that you don't spend a ton of money on: it's a service that is not a large percentage of your bill. Let's say you use Workspace, and you only spend $20 a month on Workspace. You ultimately do want to know if that spend spikes 100 percent or 200 percent, but overall, that's only maybe $20 on your bills. So, that's not something to see very easily unless it spikes exponentially. 

So, the existing solutions for this problem require a lot of hands-on work to build a solution. You either need to know what your baseline spend is in the case of AWS Budgets, or you need to perform some kind of manual analysis via custom spreadsheets or business intelligence tools. But AWS Cost Anomaly Detection kind of gets rid of a lot of those things. It allows you to look at anomalous spend as a first-class citizen within AWS.

Pete: Yeah, the other trick too, with this anomalous spending—and I've gotten really good at learning how to spell ‘anomaly’ because I've always spelled it very wrong my entire life, but in just writing the preparatory material for this, the number of times I spelled anomaly has really solved that problem for me. Now, sometimes those mature organizations, they might see that anomalous spend, maybe the day after, maybe the week after, but I've been a part of organizations who they see that spend when the bill comes. That's actually pretty common. You're not an outlier if you only identify these outliers in spend when your bill arrives. And that outlier in spend could be something like, “Wow, we changed a script, and we're doing a bunch of list requests, and wow, we're that $8,000 come from?” or, “We're testing out Amazon Aurora and we did a lot of IOs last weekend, and our estimated bill is going to be $20,000.” Those are all things that if you're not a crazy person who's so in love with your bill that you look at it every day, you're going to miss that, right? You're just going to wait to the invoice. That's what everyone happens, right, Jesse?

Jesse: Absolutely. Yeah, it has been really fascinating for us to see this pattern again and again, honestly, with some of the clients that we worked with, but also within the companies that I've worked with over the years. It's just not something that is highly thought about until finance sees the bill at the end of the month or after the end of the month, and then it becomes a retroactive conversation, or a retrospective to figure out what happened. And that's not the best way to think about this.

Pete: Yeah, exactly. I mean, the best way to save money on your bill—something we see every day—is to avoid the charge, right? Avoid those extra charges. And the way you can do that is to know of an anomaly in advance. So, one of the best parts of this feature—I can't believe it, we've made it nearly five minutes into this conversation without calling out the most impressive part of Anomaly Detection—is the fact that it's all ML-powered. Now, I know what you're thinking, that you just cringed when I said ML, it's machine learning. And I cringe whenever a company markets based on machine learning. And the rule that I have is, you need to tell me how many PhDs are on your staff before I believe you can actually do machine learning.

Jesse: [laughs].

Pete: In the Amazon case, as it turns out, I could guess that they hire quite a few PhDs, so I feel like I'm going to give them a pass on this one.

Jesse: I feel like this is going to be a fun, over-under conversation of how many PhDs were on the team that put this service together, or built the machine learning component of AWS Cost Anomaly Detection.

Pete: I'll tell you what. It's good to be more than most SaaS services, that market towards machine learning.

Jesse: Absolutely.

Pet...