AWS Morning Brief Whiteboard Confessional: The Rise and Fall of the T-Shaped Engineer
About Corey Quinn
Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.
Links
Transcript
Corey Quinn: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.
On this show, I talk an awful lot about architectural patterns that are horrifying. Let’s instead talk for a moment about something that isn’t horrifying. CHAOSSEARCH. Architecturally, they do things right. They provide a log analytics solution that separates out your storage from your compute. The data lives inside of your S3 buckets, and you can access it using APIs you’ve come to know and tolerate, through a series of containers that live next to that S3 storage. Rather than replicating massive clusters that you have to care and feed for yourself, instead, you now get to focus on just storing data, treating it like you normally would other S3 data and not replicating it, storing it on expensive disks in triplicate, and fundamentally not having to deal with the pains of running other log analytics infrastructure. Check them out today at CHAOSSEARCH.io.
For a long time now, I’ve been a believer in the idea of the T-shaped engineer. And what I mean by that is that you should be broad across a wide variety of technologies, but deep in one or two very specific areas. So, it looks a bit like a T, or an inverted T, depending upon how you wind up visualizing that. I’m describing this with words. I don’t have a whiteboard in front of me. Use your imagination, you’ll be okay. The point being is that whenever you’re working in a new environment, or on a new problem, having a broad base of technologies of which you’re aware, is incredibly useful to fall back upon. Now, the reason to be super deep in one or two areas, is that specialization is generally what lets people charge more for various services. People want to hire domain-specific expertise for an awful lot of problems that they want to get solved. So, having something that you can bring into job interviews and more or less mop the floor with people asking questions around that domain is an incredibly valuable thing to have.
But that has some other consequences too. And that’s what today’s episode of The Whiteboard Confessional is talking about. Back in my first Unix admin job, I busily began upgrading a whole lot of the infrastructure and ripping out very early Red Hat Enterprise Linux and CentOS version 4 systems and replacing them with the one true operating system, which, of course, is FreeBSD. And I had a litany of explanation as to why it was the best option, what it could do for various problems, and why there was just absolutely no comparison between FreeBSD and anything else. I could justify it super easily, and the real defense mechanism here was that people get really, really, really tired of talking to zealots, so no one kept questioning me. They just basically said, “Fine, whatever,” and got out of the way. Years later, I decided to focus on something that wasn’t an esoteric operating system to go super deep in, and that’s right, I picked SaltStack, which is configuration management done right, tied to remote execution.
I’d worked with Puppet, I’d tolerated CFEngine, but I had a bunch of angry loud opinions about it and SaltStack was absolutely the way and the light. So, in the company I was working at at that time, I rolled it out everywhere, and our entire provisioning and configuration management process was run through SaltStack. And I could come up with a whole litany of reasons why this was the right answer, and that no one else was going to be able to come close to what the ideal correctness that SaltStack provided. And people eventually stopped arguing with me, because they had better things to do than argue with a zealot about which configuration management system was the right one to go with. I’ve also talked on previous episodes of the show about using ClusterSSH. And this was before I discovered the religion that was configuration management.
It was the right answer, because rather than having to run a for loop with shell scripting, which was suboptimal for a wide variety of reasons, and I would explain to everyone why it was suboptimal. So, again, they shrugged, got out of the way and let me use ClusterSSH. And a similar pattern happened when I was working with large scale storage. NetApp was the right answer for all of our enterprise storage needs because let’s face it, it wasn’t my money. And when it comes to NFS, even today, they are head and shoulders above anything else in the space. And then eventually, it turned to AWS. And for a while, I want to say around 2014, 2015, I would tell you why AWS was the right answer for every problem. What challenge are you trying to work with? Well, AWS has an answer for that, because of course, they do. Their product strategy is, “yes”. Now, what do all of those independent stories have in common? Great question. Let’s talk about that. But first…
In the late 19th and early 20th centuries, democracy flourished around the world. This was good for most folks, but terrible for the log analytics industry because there was now a severe shortage of princesses to kidnap for ransom to pay for their ridiculous implementations. It doesn’t have to be that way. Consider CHAOSSEARCH. The data lives in your S3 buckets in your AWS accounts, and we know what that costs. You don’t have to deal with running massive piles of infrastructure to be able to query that log data with APIs you’ve come to know and tolerate, and they’re just good people to work with. Reach out to CHAOSSEARCH.io. And my thanks to them for sponsoring this incredibly depressing podcast.
The problem is, is that everything I just mentioned, was a pet technology, or a pet company. Something that I had taken the time to get deep in and learn. And therefore, it became my de facto answer for a...