Links
- Last Week In AWS Twitter: https://twitter.com/lastweekinaws
- https://wellarchitectedlabs.com/
Transcript
Corey: This episode is sponsored in part by Catchpoint. Look, 80 percent of performance and availability issues don’t occur within your application code in your data center itself. It occurs well outside those boundaries, so it’s difficult to understand what’s actually happening. What Catchpoint does is makes it easier for enterprises to detect, identify, and of course, validate how reachable their application is, and of course, how happy their users are. It helps you get visibility into reachability, availability, performance, reliability, and of course, absorbency, because we’ll throw that one in, too. And it’s used by a bunch of interesting companies you may have heard of, like, you know, Google, Verizon, Oracle—but don’t hold that against them—and many more. To learn more, visit www.catchpoint.com, and tell them Corey sent you; wait for the wince.
Pete: Hello, and welcome to the AWS Morning Brief. I am Pete Cheslock. I'm still here. I'm going to be here for a while I guess, but not alone. I'm here with Jesse. Jesse, thank you again for coming on board and keeping me company.
Jesse: Always a pleasure.
Pete: It's honestly just nice to talk to someone else that's outside of my little family unit or my pandemic crew.
Jesse: I would say it's nice to get paid to just talk about my feelings. But I mean, I'm not technically getting paid for this.
Pete: Yeah, I feel like I'm just trying to balance the conversations with coworkers, podcasting this, my kids at this point, have more Zooms than I do.
Jesse: [laugh]. I think that probably says something about our social lives and about ourselves. And I feel like I need to go rethink everything.
Pete: Well, my son who is six years old, he does a better job of managing his mute button than most full-grown adults I know.
Jesse: I feel like that's the fun thing. I really want to see how the next generation is going to grow up with technology, better understanding the mute button, and all of this video content than we do.
Pete: It is hilarious to hear my daughter yelling at her friends, “You're on mute.” [laugh]. Oh, well, what is not on mute today is both of us. We are talking about the most loved Amazon service, Amazon QuickSight.
Jesse: I think it's technically going to be on blast today rather than on mutes.
Pete: Yeah, I think we're going to struggle to keep this one on time. So, if we go long, I apologize in advance. But we're talking about QuickSight, which for those that maybe have never heard of QuickSight before, it's Amazon's business intelligence tool. The question you're probably asking yourself, to be perfectly honest, is why? Why did you even try QuickSight?
Like what point, what thing were you solving that made you think of QuickSight? So, we're going to tell that story. But first, let's just pivot into BI tools, business intelligence tools. That's the category that QuickSight is technically in. So, we'll talk a little about that, and also how we actually use BI tools within Duckbill because that'll give you, hopefully, the context into answering that question of, “Why did you even try QuickSight, Pete? Why?”
Jesse: I mean, I feel like there's probably still going to be people asking us why after this podcast, and I'm sorry for those listeners. We don't have an answer for you. Maybe we're just masochists. We don't know.
Pete: It's just because it's there, I think is what the final answer is. [laugh].
Jesse: Absolutely. So, business intelligence tools solve a whole variety of problems and we could probably do an entire episode on them in general. They help you gain insights from your data, which is fantastic. I absolutely love that this is even a category of service out there. But today specifically, to keep it on track, we want to specifically talk about gaining insights from your spend data, your AWS spend data. And to do that, we really need to start by talking about the AWS Cost and Usage Report.
Pete: Yeah, the Cost and Usage Report—you might hear it referred to as the CUR. I heard it referred to as the CUR often and it took me quite a while to actually figure out what anyone was talking about. So, if you hear someone say the CUR, they probably mean the Cost and Usage Report. But this is the v2, we'll call it, version of the Amazon billing data.
It's incredibly high fidelity, I think is the term. It's very granular; there's a lot of data in there. And it's not enabled by default; you need to actually go turn it on. But what's awesome about this tool is it can provide you some really deep insight into where your money is going, and the only cost for it is the cost to store the data. And the Cost and Usage Report itself, when you turn this report on and have it dumped into your S3 bucket location of choice, you can actually have it store into a couple different file formats.
One of them is Excel CSV format. And the other one is a Parquet format, which is a columnar data store and is a lot more efficient for this type of data. And it's the Parquet version of this that we use, we tell our clients—clients of Duckbill—to turn this on and turn it on with Parquet because then you can use tools like Athena to query your data and just leave it in S3 and run those ad hoc queries. So, Athena, though, which we're not talking about Athena, is challenging to use, in some cases—
Jesse: Yeah.
Pete: —you have to know SQL, which if you don't know SQL you're kind of in a bad spot. So, we use a BI tool, a very popular one called Tableau to query our data on Athena. So, Athena is kind of the engine, you could also obviously put your CUR data into an actual database. But largely, the queries we're doing, these are all human-generated. We're fine if they take seconds; they don't need to happen in milliseconds.
Jesse: Yeah, I mean, there's lots of solutions out there. There's third party commercial apps like Tableau and Looker—RIP—there's open-source options like Metabase. But of course then, in true AWS fashion, there's also a hastily integrated acquisition called QuickSight.
Pete: So, I have this memory in my head—and hopefully someone will correct me if they're listening to it, and I'm wrong here—but I feel like QuickSight was actually an acquisition. Like Amazon, which really doesn't usually acquire a lot of teams or businesses into Amazon Web Services, with like a couple of pretty rare exceptions, I'm almost positive, that QuickSight was actually some other product that Amazon acquired into it. But the history of QuickSight from at least the Amazon umbrella started around 2015 is when they announced it at re:Invent, and I was there for that announcement. I remember that announcement clearly, and I still actually kind of laugh at it when it came out. Now, first off, that was 2015 is when it was announced, and not for nothing, it does not look like it has gotten much better in the five years that it's been operating since launch.
<...