Humans Are the Most Expensive Part of Cloud

Links:

• Unconventional Guide to AWS Cost Management: https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/

Transcript

Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.

Corey: Ever notice how security tends to be one of those things that isn’t particularly welcoming to folks who don’t already have the word ‘security’ somewhere in their job title? Introducing our fix to that, Meanwhile in Security. To sign up for the newsletter or to find the podcast, visit meanwhileinsecurity.com. coming soon from The Duckbill Group.

Pete: Hello, and welcome to Fridays From the Field. I'm Pete Cheslock.

Jesse: I'm Jesse DeRose.

Pete: And we're back, again. We're continuing our series, the Unconventional Guide to AWS Cost Management. And as always, if you have questions, as we are going through this series and want to learn more, go to lastweekinaws.com/QA. Thank you to all of those who have already submitted questions. 

Jesse: Yes.

Pete: Really great ones coming in. 

Jesse: Thank you.

Pete: We're going to take a couple of episodes in the future to answer those questions and really dive into them. So, keep them coming. We really love them so far. So Jesse, what are we talking about today?

Jesse: Today, we're going to be talking about one of my favorite topics, which is that humans are the most expensive part of Cloud. 

Pete: Yeah, we hear this quite a bit. I mean, not just in salary, right? This is the line that usually is mentioned when we talk to folks about their Amazon spend. They say, “Well, outside of salary, Amazon is our most expensive bill.” 

Jesse: Yeah.

Pete: That line has been repeated more times than I can count.

Jesse: But what's so fascinating to me is that this really gets at the idea of total cost of ownership. I think that's ultimately what I really want to focus on for just a second. Total cost of ownership is thinking about all of the spend related to your cloud costs. Now, when you think about cloud costs, you will generally think about just the usage that you have within AWS, maybe some discounts from either an EDP or PPAs. But are you thinking about how much time it's taking your engineers to manage all of that usage, manage that infrastructure, manage the deployment pipelines that are living within the cloud? Are you thinking about all of those components and the cost of those components alongside your usage?

Pete: Yeah, exactly. I think engineers are bad at this.

Jesse: Yeah.

Pete: Myself included. But this is something where we want to build things. That's why we're in this industry. And it's fun to build things. Maybe not so much fun to, kind of, ongoing manage those things. Looking at you, Cassandra and Elasticsearch clusters.

Jesse: [laugh]. Yeah, it's this idea that there are definitely opportunities for engineers to spin things up and manage things on their own when you want to build that Kubernetes cluster and learn how to manage a Kubernetes cluster, learn how to build a Kubernetes cluster. That's great. We don't want to stop you from building and learning at all. But when you're building infrastructure for your organization, for your teams, for your products, is it going to be more cost-effective for you to build this solution yourself, or is it going to be more cost-effective for you to leverage existing managed services within the cloud?

Pete: I like to call it operational FOMO, you know, the fear of missing out. And I think a lot of engineers suffer that when it comes to the new hotness, the new stuff. Kubernetes is a great example. I mean, I feel like a lot of those people were also equally like, “OpenStack is going to be the best thing ever.” And then it didn't. 

But I like to think of my time at a previous company where we deployed into the Cloud, specifically Amazon, and there was a fear that was, again, we've mentioned this before, it's an irrational fear about vendor lock-in. And that fear forced us into building forced us only using core primitives: S3, EC2, EBS, really. We really didn't use much more than that. I mean, obviously, the networks and stuff go in there. And the idea was, is that oh, well, we have this portability. 

And we—Duckbill Group, Corey, we've all talked about it, written about this. It's a fallacy. You're locked in for a lot of other reasons that I'm not going to go into right now. But because of that, we became very good at running our own databases and specifically consuming a large amount of time-series data. It was a security event application. 

And so one of the interesting flip sides of this outcome is that we ran our own monitoring infrastructure. I didn't pay for Datadog. They called me every single day and I was like, “My metrics infrastructure cost me $1,000 a month. You're going to charge me $50,000 a month. Even if you discounted that by half, I still am going to pay a lot more.” 

And the reality was, is that we became so good at managing these systems, we didn't need those services. But I always think back at like, at what cost? How much more time could we have invested in the application, the product, how we deployed it, availability, all that stuff, if we hadn't had to invest so much time into running our own Elasticsearch, running our own Mongo, our own Redis, our own Cassandra? We spent a lot of time doing those things.

Jesse: Yeah, there's a lot of opportunities to leverage managed solutions for those things. Because, again, part of it is this idea of your engineers don't have to spend time managing this infrastructure; they can spend time on other things. But also think about what are the other cost components of this architecture that you may be able to leverage by using a native or a managed AWS service? For example, if you look at Amazon Elasticsearch—is it ‘Amazon Elasticsearch?’ Is it—

Pete: I always forget if it's ‘Amazon Elasticsearch’ or ‘AWS Elasticsearch.’ And oftentimes, it doesn't feel like a rhyme or reason why they name it the way they do.

Jesse: Well, let me put it this way. If you look at the managed Elasticsearch service on AWS, you don't end up paying for some of the things that you might pay for if you were managing that infrastructure yourself, like data t...