AWS Cost Anomaly Detection 2: Electric Boogaloo

About Corey Quinn

Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.

Transcript
Corey: This episode is sponsored in part by Catchpoint. Look, 80 percent of performance and availability issues don’t occur within your application code in your data center itself. It occurs well outside those boundaries, so it’s difficult to understand what’s actually happening. What Catchpoint does is makes it easier for enterprises to detect, identify, and of course, validate how reachable their application is, and of course, how happy their users are. It helps you get visibility into reachability, availability, performance, reliability, and of course, absorbency, because we’ll throw that one in, too. And it’s used by a bunch of interesting companies you may have heard of, like, you know, Google, Verizon, Oracle—but don’t hold that against them—and many more. To learn more, visit www.catchpoint.com, and tell them Corey sent you; wait for the wince.

Pete: Hello, and welcome again to the AWS Morning Brief: Whiteboard Confessional. Corey is still enjoying some wonderful family time with his new addition, so you're still stuck with me, Pete Cheslock. But I am not alone. I have been joined yet again, with my colleague, Jesse DeRose. Welcome back, Jesse.

Jesse: Thank you for having me. I will continue to be here until Corey kicks me back off the podcast whenever he returns and figures out that I've locked him out of his office.

Pete: We'll just change all the passwords and that'll just solve the problem.

Jesse: Perfect.

Pete: What we're talking about today is the “AWS Cost Anomaly Detection, Part Two: Electric Boogaloo.”

Jesse: Ohh, Electric Boogaloo. I like that. Remind me what that's from. I feel like I've heard that before.

Pete: Okay, so I actually went to go look it up because all I remembered was that there was, like, a movie from the past, “Something Two: Electric Boogaloo,” and I dove to the internet—also known as Wikipedia—and I found it it was a movie called Breakin’ 2: Electric Boogaloo], which is a 1984 film. And it says it's a sequel to the 1984 breakdancing film Breakin’: Electric Boogaloo, which I thought was kind of interesting because I always thought of that joke ‘Electric Boogaloo’ was as related to the part two of something, but it turns out it's not. It's actually can be used for both part one and part two.

Jesse: I feel like I'm a little disappointed, but now I also have a breakdancing movie from the ’80s to go watch after this podcast.

Pete: Absolutely. If this does not get added to your Netflix list, I just—I don't even want to know you anymore.

Jesse: [laughs].

Pete: What's interesting, though, is that there was a sequel called Rappin’, which says, “Also known as Breakdance 3: Electric Boogalee.”

Jesse: Okay, now I just feel like they're grasping at straws.

Pete: I wonder if that was also a 1984 film. Like, if all of these came out in the same year. I haven't looked that deep yet.

Jesse: I feel like that's a marketing ploy, that somebody literally just sat down and wrote all of these together at once, and then started making the films after the fact.

Pete: Exactly. One last point here, because it's too good not to mention, was that it basically says that all these movies, or at least the later one, had an unconnected plot and different lead characters; only Ice-T featured in all three films, which then got me to think a sec—wait a second, Ice-T was in this movie? Why have I not watched this movie?

Jesse: Yeah. This sounds like an immediate cult classic. I need to go watch this immediately after this podcast; you need to go watch this.

Pete: Exactly. So, anyway, that's the short diversion from our, “AWS Cost Anomaly Detection, Part Two” discussion. So, what did we do last time? Why is this a part two? Hopefully, you have listened to our part one. It was, I thought, quite amazing—but I'm a little bit biased on that one—where we talked about a new service that was very recently announced at Amazon called AWS Cost Anomaly Detection. 

And this is a free—free service, which is pretty rare in the Amazon ecosystem—that can help you identify anomalies in your spend. So, we got a bit of a preview from some of the Amazon account product owners for this Cost Anomaly Detection, and then we got a chance to just dive into it when it turned on a few weeks ago. And it was pretty basic. 

It's a basic beta service—they actually list it as beta—and the idea behind this is that it will let you know when you have anomalies in your cost data, primarily increases in your cost data. I remember specifically talking that it was specifically hard to identify decreases in spend as an anomaly. So, right now it only supports increases. So, a few weeks ago, we went into our Duckbill production accounts, turned it on, and we were just waiting for anomalies so that we could do this.

Jesse: I also think it's worth noting that I'm actually kind of okay with it being basic for now because if you look at almost any AWS service that exists right now, I would say none of them are basic. So, this is a good place to start and gives AWS opportunities to make it better from here without making it convoluted or difficult to set up in the first place.

Pete: A basic Amazon service, much like myself.

Jesse: [laughs].

Pete: So, guess what? We found anomalies. Well, we didn't find them. The ML backing Cost Anomaly Detection found some anomalies. So, that's what we're here to talk about because now that we actually have some real data, and real things happened, and we actually dove into some of those anomalies, interestingly enough. So, that's what we're here to talk about today.

Jesse: It's also probably worth noting that we changed our setup a few times over the course of kicking the tires on this service, and unfortunately, we weren't able to thoroughly test all of the different features that we wanted to test before this recording. So, we do still have some follow up items that we'll talk about at the end of this session. But we did get a chance to look at the majority of options and features of this service, and we'll talk about those today.

Pete: So, if you remember—or maybe you don't because you didn't listen to the last episode we did—we configured a monitor, is what it's called, that will analyze your account based on a few different criteria. And the main one is,...