Geeking Out with Adriana Villela

About our guest:Developer Advocate, teacher, and failed startup founder. Published author. Currently leading all things DevRel at Tracetest.io.Find our guest on:bento.me/adnanrahic (link to Adnan's socials)Adnan on MediumFind us on:All of our social channels are on bento.me/geekingoutAll of Adriana's social channels are on bento.me/adrianamvillelaShow Links:ViberNode.jsAngular JSJQueryJava SwingFreeCodeCampTracetestMalabiAspectoJest and AVA (JS testing tools)JUnitCypress (testing)Playwright (testing)K6 (load testing)OpenTelemetry DemoTracetest in the OpenTelemetry DemoCloud Native Live: The power of traces - why OpenTelemetry embraced trace-based testingGoogle Hipster ShopTrace-based Testing the OpenTelemetry Demo by Daniel DiasTracetest on Nomad (Adriana's GitHub)Tracetest Cypress IntegrationTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out. The podcast about all geeky aspects of software delivery DevOps Observability, reliability and everything in between. I'm your host Adriana Villela coming to you from Toronto, Canada. And geeking out with me today. And you know what? I did a really crappy job because I forgot to ask how to pronounce your last name.ADNAN: It's so difficult that you'll butcher it. I don't mind. Seriously, take a stab and try your best.ADRIANA: Okay. You're not even going to give me a hint?ADNAN: Go for it.ADRIANA: Oh, dude. That's mean!ADNAN: The hint is I am from the Balkans. So eastern Europe, That's the only hint?ADRIANA: Yeah, so exactly. I don't know how to pronounce the funky accents any more than most people don't know how to pronounce the weird brazilian accents. So basically geeking out with me, I have Adnan, whose last name I cannot pronounce. No, I'm going to try. Hang on. That's so mean.ADNAN: Hey, everyone. Yeah, my last name is pronounced "rah-hitch", which is the c with the thing on it (ć) is like a "ch" sound. So it's basically if you're eastern European, you'll basically know if you're anything else you would have no absolute clue. Also funky because my name is Adnan, which is Arabic and I'm white, so that's kind of strange as well. But yeah, that's just being Bosnian, you know? We have a lot of different cultures here in the Balkans and yeah...super, super proud of that and happy to be here as well. So I've known Adriana for a couple of years now.ADRIANA: We are KubeCon buddies!ADNAN: Yeah.ADRIANA: Yeah. I'm actually super excited to meet up with you at Kubecon in Paris.ADNAN: Same, same. It's going to be great. I've never been to Paris.ADRIANA: I went when I was a kid with my parents so I was 18 or 19, I think, at the time. And it's different when you travel with your parents. So I guess I wasn't like a kid kid, but I traveled with my parents so it is going to be a different experience compared to not traveling with my parents. But I mean when I went I enjoyed the touristy things. I'm hoping I can do some more kind of like off the beaten path in Paris.ADNAN: We need to figure out anybody that speaks French and just meet up with them so they can take us around.ADRIANA: Oooh, I speak French.ADNAN: Oh, right, Canada!ADRIANA: Canada. Yeah, my high school...okay... Canada is technically a bilingual country and I will put it in air quotes because, yes, we are a bilingual country, and you look at the packaging for all of our products, and it's bilingual, but realistically speaking, the further west you move from Québec, the less French you speak in school. So, like, it's mandatory in elementary school, but it peters off the further west you move. So I was lucky enough to have gone to high school in Ottawa, Canada's capital, which is a very bilingual city. And so my high school had, like, kind of a...they had an immersion program where they had different levels of immersion. So you could do, like, full immersion or half immersion. So in high school, I did half immersion for, like, I think, two years. So it meant, like, half of my classes were in French, and one of my classes that was in French was gym class.ADNAN: Wow, that was a smart move, right?ADRIANA: Yeah. But for me, picking up French wasn't so hard. I moved to Canada when I was ten, and picking up French wasn't terrible because I speak Portuguese. And so I found a lot of parallels between French and Portuguese.ADNAN: Latin languages are easier, right, if you speak one Latin language, picking up another one. Same with the Slavic languages here in eastern Europe. I have a friend who's from Czechia, and she picked up Bosnian in one year, and it's crazy. Having a Bosnian boyfriend helped her as well.ADRIANA: Yeah.ADNAN: I'm not going to lie, but still, it's doable, right? So if you speak any language that, any language from Russian to anything here in the Balkans, it's so similar that you can pick up a lot. So I imagine it's the same. I have actually two of my coworkers. One is Brazilian and one is Argentinian. But this Brazilian guy lives in Argentina and he speaks Spanish and Brazili- sorry. What the hell? Portuguese. He speaks Portuguese and he switches between them fluently and seamlessly. Right? So I think that's really cool.ADRIANA: That's very cool. I find, like, as a Portuguese speaker, understanding most Spanish words is fairly straightforward. So, like, when I went to Barcelona in 2019, I spoke "Portoñol", so I just threw in some words in Portuguese with a Spanish-ish accent and prayed that I was understood. And I went into a shoe shop in Barcelona, and I was able to actually buy a pair of shoes speaking, my second-rate Spanish.ADNAN: That happened to me in Slovakia. I was. I was in Slovakia with my wife a few months ago and went into a store, and we kind of figured it out. They knew zero English, for the record. We kind of had to do the Bosnian Slovakian and back and forth, and it went surprisingly well. So I'm very proud of that. I'm very happy to see that it works for you as well.ADRIANA: Yes. That's so super cool. Well, are you ready to get into some of our lightning round questions? All right. Okay, let's do this. Question number one. Are you left handed or right handed?ADNAN: I am right handed. So falling while skiing and hurting my left is not a problem. Eating with utensils is an issue because...that was a problem. So I kind of have to do this old switcheroo, but yeah.ADRIANA: Next question. Do you prefer iPhone or Android?ADNAN: I was an avid Android user up until, was it probably two, three years ago, and I switched everything over to Apple products. Phone, Macbooks, the headphones, earphones. Actually, I only have the Bose noise canceling headphones that aren't Apple. Everything else is basically Apple, which is...My wife hates it because she runs Android, so I can't airdrop her photos. So I need to buy the family package iCloud, so I need to send her links. And she doesn't want an iPhone. So yeah.ADRIANA: I feel ya. My sister is...she's an iPhone user, but her husband is like, an Android user, so then she's always like, I don't know how to get files over to him. I'm like, Google Drive?ADNAN: And then she goes like, just send me on Viber. It's like it destroys the photo. If I send you a photo, we go to some beautiful place in freaking Dubrovnik where they did the Game of Thrones thing, and I take a picture, she'll, "Just send me on Viber." Like, it's going to destroy the photo. And she doesn't get it. She's not an IT. Like, she doesn't get it. It's frustrating.ADRIANA: These are the issues that divide families. Just kidding.ADNAN: iCloud family solves everything, right?ADRIANA: Yeah, it's the age-old battle. My husband and I converted to Mac. Jeez...we were, I think, like four or five years married. Before that. We're like, "No! Windows! No Mac!" Like Linux, fine, but Windows or whatever, right? And we were BlackBerry users, and Mac was like...ahhhh...And then for funsies, I got...it was a Mac Mini. I think one of the first years, the Mac Mini came out and they were like, super cheap.I'm like, let's use this as a media server. Then we plug it in, and I'm like, ahhhh, I don't know how to use this. And then for funsies, I'm like, I'm going to get myself a MacBook Pro because it was like a Black Friday sale or whatever, which in Apple terms, Black Friday sales are like, it's the saddest sale ever because you're not getting a deep discount. But I'm like, oh, discount on Apple product. And then I started using my MacBook Pro, and I just bought myself like this Core i7 Intel machine super beefed up. And I'm like, but I want to use my Mac Pro, which isn't nearly as beefy. And it's so beautiful. That was kind of it for me, that converted me from then on and was like, yeah, I can't go back.ADNAN: It's too good. It just works.ADRIANA: Right? Yeah, it just works. I'm with you. Okay, next question. Similar vein. Mac, Windows or Linux?ADNAN: Oh, I was super into Linux up until I got my Mac. I was only running Linux and I just switched to Mac when the M1 chip got released. That was quite recent as well. So I was running full on Linux up until that. Absolutely hated Windows. Nothing against Microsoft. I think they're great as a company and great with what they're doing. And Windows as a system...it's perfectly fine. I just couldn't...it was just so horrible. But yeah, no, it is Mac all the way right now.ADRIANA: I feel you. Yeah. And the M chips are like..."MWAH!" When I went from my Intel Mac to my M1 Mac, I'm like, I don't hear a fan. My computer is not burning my lap. What's going on?ADNAN: And I'm still using the Air one, which is the small one without the fans, without the anything. And you charge it like a phone, you plug it in overnight.ADRIANA: Oh, yeah, the charger is like tiny.ADNAN: Plug it in overnight, you take it off the charger in the morning, you use it all day. You get back home, you plug it in when you go to sleep, and you never charge it. It's nuts.ADRIANA: The M1 Mac airs are so pretty. It's like a little piece of jewelry for your desk.ADNAN: And it goes with my very small hands. I have very ladylike hands. Right. The palm is super small, so it kind of fits nice. Right. But I need to get a bigger one. I need to get a proper one. It's getting a bit crowded with all of the Kubernetes stuff I need to run and demo.ADRIANA: Oh, yeah, I feel you. I went for the 14 inch instead of the 16 inch M1 Mac Pro...MacBook Pro. Because even though I like the screen size on the 16 inch, it's so much thicker compared to the Intel Macs of the same size that I was like. And I had a 15 inch Intel Mac. I was like, holy crap. It just feels too big. But to each their own. That's just my personal...ADNAN: I'm actually not against them bringing back all of the ports. It's fine that it's a bit thicker with all those ports. I'm right now connected with a dongle that's massive and it's taking up a ton of space on my table with all of these microphones and whatnot. And it would just make more sense to just have the jack. It's just like why not, right?ADRIANA: Yeah, I agree. It's nice having the HDMI back on the computer. And I think I have like an SD slot. Yeah, looking over, I have an SD slot on my Mac.ADNAN: You're a podcaster, you need that, right?ADRIANA: I know, right? Yeah, it's true, it's true. Okay, next question. Favorite programming language.ADNAN: Is Javascript even a language or is it all fake? What do you think?ADRIANA: Don't ask me about JavaScript. My opinions are as strong as using Windows as a primary machine. I don't want to offend anyone who likes either of those things, just a personal preference.ADNAN: I mean, I love it. I've used Javascript for ages. You remember when Angular JS 1.0 was released? Like Angular JS? That was like freaking ten years ago now. That's when I use Javascript on the front end, in Javascript terms old. I was around when Node was 0.12. That's when I started using Node.ADRIANA: Wow, that's old school. Dude, but I am older because I remember when Javascript came out before Angular and Node. Actually, and it was the OG Javascript that really turned me off from Javascript and then I ran away screaming.ADNAN: Oh, JQuery was terrible. JQuery was the bane of my existence. It was horrible.ADRIANA: I vaguely remember jquery, I just remember trying out a few things in Javascript and being like, "Next, please. I'll go to backend development, thank you." Ran to my Java. Now mind you, Java had like a front-end thing, I think it was called Swift (NOTE: it was Swing) and it was like a piece of crap. Also ran away.ADNAN: I would have thought that Java wouldn't be good in the front-end.ADRIANA: Yeah, it was bulky and horrible. Next question. Do you prefer dev or ops?ADNAN: Tough one, tough one. I do prefer dev. I do prefer dev, but I've been tightly linked to ops for five or so years, so it's a tough one, but still, I would still prefer dev. I don't mind doing the ops, but I think the automation part of ops I really like. The Kubernetes part of ops is painful. I'm not smart enough for that stuff. I swear to God. Networking and all of those infra Kubernetes things, I'm definitely not smart enough for that.ADRIANA: Kubernetes is definitely a test of patience. There's like so many aspects to it that I feel like you can be really good in one area of it and okay-ish in other areas. Enough to be dangerous kind of thing.ADNAN: Yeah, but then there are so many tools. There are so many things that you need to know. I have a buddy, actually, my best man. He's been doing ops...so automation and basically all things Kubernetes for years now. And dude, I ask him something and dude pulls out an answer out of thin air and then tells me something I've never heard about. And he goes like. Says something like, "Did you taint?" What are you talking about? It's like a tribal thing. It's tribal knowledge, that stuff. I don't really know where people learn all that. So...crazy.ADRIANA: I think it's usually like learning out of necessity. I find all my Kubernetes knowledge is learned out of necessity. It's like I need to figure out this thing. But hey, it means that you basically have a Kubernetes zombie apocalypse friend that if there's like the Kubernetes zombie apocalypse, you know who to call for help.ADNAN: Oh, yeah, for sure. He's the one. What's it called? He works in a platform team. And the platform team is all just developer experience, performance, reliability, uptime. So he's the person you call when shit hits the fan.ADRIANA: That's awesome. Okay, next question. Do you prefer JSON or YAML?ADNAN: Oooh...YAML. YAML.ADRIANA: YAML. Yeah, I'm Team YAML.ADNAN: Who doesn't love YAML?ADRIANA: The JSON people. So many people bitch about YAML. And yes, there are annoying things about YAML.ADNAN: I like to call myself a "full stack YAML developer".ADRIANA: I feel you. Lots of time buried in YAML. Okay, next question. Spaces or tabs? Which one do you like better?ADNAN: It's a tough one because I set up VSCode to do spaces, but I hit the tab and then it does the spaces.ADRIANA: Yes. Hey, that's how mine's set up too.ADNAN: Yeah, big brain move.ADRIANA: Yeah, I used to be like all tabs, but I don't know, I feel like YAML kind of made me do it. For some reason. It was an incentive to convert my tabs to spaces as you do. You hit tab and it turns it into spaces. Magic. Okay, two more questions. Do you prefer to consume content through video or text?ADNAN: For sure. I don't mind videos. It's just that it's either really long video. I mean, really long. What's really long nowadays? Ten minutes and above.ADRIANA: I know, right? TikTok nation.ADNAN: Three minutes and above, people think it's long nowadays. I'd say ten minutes. Above that is probably videos that I like watching just because of the content I can get out of it and learn from it. And then otherwise, just for day to day stuff, I would rather read a docs page, I would rather read a blog post than I would watch a video, which is, I've heard pretty strange people don't agree with me, so I might be. Yes.ADRIANA: Really? That's so funny. Because most...So then I have a very interesting, like, I have a skewed population then on this podcast, because most of the folks I've asked this question to all lean towards text.ADNAN: We're the old guard.ADRIANA: Yeah, I know, right?ADNAN: Born and raised. Before Facebook, before Instagram, TikTok.ADRIANA: I know. It's true. It's true. Yeah, I know. Whenever I watch old TV shows and there's, like, no cell phones and they're, like, calling from a pay phone, I'm like, "Just text them!" Obviously very tongue-in-cheek comment. But I'm like, damn, we are used to things that...the things that we're used to now, we take so much for granted.ADNAN: Oh, yeah, for sure.ADRIANA: Okay, final question. What is your superpower?ADNAN: Ooh, my amazing sense of humor.ADRIANA: Dun dun, dun.ADNAN: I didn't. That was terrible. That was terrible. I don't really know. I would say I am moderately funny. That would be it.ADRIANA: Which is a good superpower because you have to break the ice. Right? Especially when you're dealing with people, which you have to as part of your job. And so being moderately funny to kind of crack some exterior shells of grumpy. When you're interacting with certain people, I think that's a skill.ADNAN: I also like listening. So I like hearing something interesting, and I'm very particular with the people I have in my friends group where I generally like hearing something that I can relate to and also think about. So I like hearing that from people I don't know as well, because hearing something that you can think about and can motivate you or you can just have that thought, oh, that's really cool. Let me brainstorm on that and then figure out something that can benefit myself from that and then give feedback to the person and then have a conversation about that. I think that's where especially for me, I like the listening aspect of that. I think that's really cool.ADRIANA: Yeah, for sure. Totally agree. All right, well, you survived the lightning round.ADNAN: Yeah.ADRIANA: Give yourself pat on the back.ADNAN: Not that bad. You told me it's going to be horrible. It was not that bad.ADRIANA: I didn't say it was going to be horrible. Some people get nervous. It's supposed to be an icebreaker. Okay, well, now that we've cleared the lightning round, questions, why don't you tell folks about yourself? Like, what do you do?ADNAN: Yeah, I mean, the quick rundown would be that I do developer relations. So I've been doing developer relations since it was called technology evangelism. So I've been doing it for a while. And yeah, I think it is awesome that I am quite literally an influencer for tech people, which is very strange. I'm not very good at influencing or influencing or whatever the word is, but yeah, I like talking to people, I like listening and I like educating. And I think that was the reason why I moved into developer relations from being an engineer...so I was an engineer for almost half my career so far, and then was basically either do developer relations, which I ended up being very good at, or be a mediocre engineer. And I was like, yeah, let's kind of try doing the thing where I'm actually kind of better, instead of being kind of average as an engineer.So I said, yeah, what the hell? Why not let me try this. And that was back when the Medium blog was taking off. They had just launched their, what's it called? They had publications and whatnot, and they just taking off. Oh yeah, FreeCodeCamp was a thing back then. So I hosted a, I mean, still is now, but back then we were just starting out all over the world. They had local groups that were doing meetups, events, education, teaching people, and I was doing a lot of in-person education as well, just teaching courses. That's basically just because a startup I was doing early in my career was all focused on education as well, just like generating courses, just like Udemy, but for a smaller local market. And that kind of fell through.And then I just started doing education myself and then doing the FreeCodeCamp thing. And then the medium thing took off and my blog on Medium just got crazy. It took off so strangely that I got like 5,000 followers within less than a year just because of FreeCodeCamp's publication taking off and publishing. Yeah, I didn't really know what to do. This 24 year old kid, just like having a medium thing take off, what the hell? So that was super fun. And then I did a lot of FreeCodeCamp, was leading the local FreeCodeCamp community as well for a couple of years and that was when I figured, well this is fun. So I took a DevRel job and yeah, I've been doing it ever since and it's basically always been something in the Observability monitoring, log management, I'm going to say...space.I did some big data as well, one previous job, but it's always been something like this...that's, I'm going to say, a niche market that isn't really easy to figure out and that requires a lot of handholding and a lot of help from the, I'm going to say marketing support, sales and DevRel teams, especially for building products. And I had a nice foundation in product development because I had a startup early in my career, so I had a touch of, Okay, so the code you're writing isn't...it's not what you write and the code you write, it's the product that you generate. Nobody cares about the code, people care about the product you're putting out, right? So I think that mindset early in my career helped me a lot with DevRel as well. And yeah, I'm carrying that with me and I think that's the, if anybody asks me for advice regarding how to be good at developer relations, it's just you have to put out a good product and you have to be the connection between the end user and your team that builds the product and that's it.If you want to have product-led growth, if you want to build a community, if you want to build the influence of you as a person, you need to have a product that people want to use and you're the person that needs to tell people if it's usable or not. And by people I mean your team. I think that's the only golden rule, if there is any golden rule in developer relations is you have to be very realistic. I make jokes about us being influencers and us doing popularity contests and we do Only Fans for engineers. It's all jokes. I'm not serious. Those are humor, like comedy. But truthfully that's irrelevant.Truthfully, it's all about the product and it's all about you figuring out a strategy to position the product in a way that makes sense and for you to position yourself as somebody in the industry that is influential to talk about the industry or the thing your product is doing. So yeah, this got super serious super quickly.ADRIANA: But I think you make a really good point, because being in developer relations, you have to build trust between your audience...between you and your audience, right? They have to know that you're not like just some sleazy ass salesperson who's just trying to sell them on a product. They need to believe that you have something interesting to say and that, oh, by the way, I represent this product as well. And because they like what you have to say, I feel kind of, it naturally gravitates towards, oh, take a look at what they do kind of thing, right? So I do feel like there's a little bit of that.I think basically making a connection with your intended audience, right? Makes a huge difference. This is not the job for people who are extremely introverted or introverts who have no desire to put on an extroverted face for a limited period of time every day.ADNAN: Kind of, I mean, and also the core or the root of our job, developer relations. It started as sales engineering at Microsoft. Or where was it started as sales engineering 15 years ago where they figured out engineers don't really want to listen to salespeople. So they took engineers and made them into salespeople and then they figured that worked back then. And then how it evolved, it didn't really work. And then they figured out the evangelist role or the technology evangelist role, which was a thing ten years ago when I first heard about it. And then that kind of stopped working because evangelist is, nobody knows what the hell you're doing. One time, one lady asked me if I do churches. Like, I swear to God. She was like, what do you do? I was like, I'm a technology evangelist. Oh, so you do like church? I was like, no, I don't do the church. Like, what the hell? I was like, probably that's the problem. That's why they changed. It doesn't really make sense. Advocate. Developer relations and developer advocate kind of sounds more normal, I'm going to say.ADRIANA: Yeah, that's so funny. I would have never thought that. But I think us being in tech, it's not something that we would have necessarily associated with church. I could see how that...yeah...You were saying that you're working in an area that's fairly niche, right? Like Observability. And it's definitely one of those areas that has expanded a fair bit in the last several years, which is super exciting. Observability is near and dear to our hearts because I think it's really the part of, it's an evolutionary step in SRE, right?You can't be a good SRE these days without having a properly observable system. So it's very exciting what's happening in the space now. So many different innovations, and especially like where you work, right? With Tracetest, I think is very exciting because of the nature of what you guys do. If you can talk a little bit about that.ADNAN: Yeah, for sure. I mean, doubling down on that Observability as a space is massive. I mean, just take a look at Datadog and their IPO. That alone is insane with regards to how much it's needed in our industry. We're not really building monoliths anymore. Like 20 years ago, not having intricate monitoring and Observability tooling was okay. It was fine. Now, what are we building nowadays? Do you think Netflix, like Netflix is thousands and thousands of services that are all interconnected.They need to communicate, they need to do something, and they're all doing it together. How do you fix that if it's broken? So that's the thing. Observability now is, I'm going to say, at the birthplace of where it's going to be ten years from now. OpenTelemetry right now, it's getting to that stable state where it's available to actually use reliably in production. More and more of the libraries are stable. More and more, I'm going to say both the metrics and logs, not just the tracing libraries, are also getting to a stable state. And a lot of the tooling around the Observability space is defaulting to use OpenTelemetry libraries and OpenTelemetry as the standard for both ingesting data, collecting, generating, and all that fancy words that we all know.And I love that. And that's where I'm thinking. We are generating huge amounts of data with traces, with distributed tracing, because of our systems being distributed. So we need to use distributed tracing to actually get a context of what the hell is happening in our system. An API call isn't really an API call. With 200, it works. An API call is an API call that calls a queue, that calls a message bus, that calls a cache, that calls a database. Those are seven different steps in one API call.So it's not really just an API call that you need to have visibility into and see what's happening. That's when I'm seeing that people are only using all of that data currently for production environments, for figuring out when their users have problems, and how to circle back to their engineering teams to actually know exactly what went wrong. Which is awesome. But that's just step one in that whole cycle. We're talking about DevOps. We're talking about that new principle of DevOps where we have a cycle of, we have the developer pushing code to the ops person and then the ops person pushes it back because they have Observability to tell the developer what's going wrong. But that cycle isn't complete without the testing part. And that's where Tracetest, where I'm working right now, we're building a tool that taps into that DevOps cycle where you're using the Observability, so the distributed tracing and all of that telemetry you get from your system from OpenTelemetry and whatever tracing backends you're using, ranging from Datadog to Sumo Logic to all of the fancy big ones to ServiceNow as well. And you're tapping into that data to run your integration testing, your end-to-end testing, your UI testing with all of these. So you're basically enhancing all of the tests you already have. So that's what I think is super cool with Tracetest, is that it doesn't just give you the test tool to tap into OpenTelemetry and run test specs on the trace data itself, which means that you can basically run a test on every single part of your transaction that an API makes. You can say, say, oh, I want to make sure that this external API call returned 200 and there is no freaking way I can do that with any test tool right now. I have to mock stuff and I have to kind of fake stuff out. I need to figure out how it works. And I spend days on that instead of let me ping the API, get real data from the trace, I write my specs on that data, that's real data and I put that in my CI and you're done.You already have the data, like freaking use it, right? That's the magical part where we're already generating all of that data, you're already keeping the data, use it for testing as well. And I think that's where I'm going to say next groundbreaking step in this DevOps cycle is going to be where test tools are just lagging behind. So we need to figure something out and hopefully Tracetest is going to fill that gap, fill the shoes or whatever we want to say.ADRIANA: Yeah, I think that's what I love the most about the idea of trace-based testing is like you're already emitting traces, just take advantage of the data. You have data, as you said, use it. And the other thing that you mentioned, which I think is something that it seems so obvious when you say it, but it's not something that we're in the habit of doing yet, which is like viewing Observability as part of the SDLC, because everyone's like, oh, it's part of an SRE practice. Absolutely. But you can't have an observable system if you don't instrument your code. Where does application instrumentation come from? During development. And therefore it means that Observability starting way earlier in the SDLC than we care to admit, right? Shifting those conversations in that direction I think will be very important really for organizations to really make the most out of Observability, right?For starters, there's so many different aspects, but instrumentation, like getting into the habit of instrumenting your code and admitting that it starts earlier, I think is so important.ADNAN: I think the main thing that people can think of from a logical point of view is that we've been instrumenting with logs for decades. I don't see anybody complaining. Oh my God, I have to add, logging, it's just normal. Why is tracing and instrumenting your code? Why is that any different? The logical aspect of that is the exact same, with one addition where you have this context that propagates. And if you use OpenTelemetry...ADRIANA: That ties everything together. Heaven forbid!ADNAN: You're using OpenTelemetry libraries anyway, so it just does it automatically. You don't really have to be any... like, you can be an average engineer as I, and I can make it work, right? So it's not rocket science. The people that made OpenTelemetry, the maintainers, they're the rocket scientists. You can just drive the rocket, you don't really need to think about how it works, right?ADRIANA: Yeah, absolutely. Yeah, I'm definitely very excited. I think when Tracetest came on the market, I think one of the predecessors was Malabi, that I think was created by Aspecto and that was just for JavaScript, like code written in JavaScript. It's really cool that now we've got a tool like Tracetest that is language agnostic and it makes sense, right? Because OpenTelemetry...you can instrument your code using OpenTelemetry in so many different languages. So it makes sense that it's not restricted to one particular language because that way we take that full advantage of being able to really instrument across microservices that are written in different languages, right? Like you said, you don't have to futz around with different mocking and different libraries, like different testing libraries for different languages. We all speak the same language and it's called OpenTelemetry.ADNAN: That's the thing where a lot of the bottlenecks when testing is just, it takes too much time. We can't really see value from all of the time that we need to put into it. Then you have the blocker of oh, I need to know JavaScript for whatever...Jest, AVA, or whatever testing tool I need. Oh, I need to use Python for whatever test tool I need to use for Python and then JUnit or whatever Java. So you're zoned in and you're kind of blocked off. You're siloed into that environment versus using trace-based testing. And using testing in general with distributed tracing with trace test is that it doesn't care about the language, it only cares about the trace. So triggering the test itself, trace test handles that through a definition that you define.The definition could be YAML. You can just click it in the UI and it triggers a test for you. You get the traces back and then you create your specs based on the traces. Now these specs, they're language agnostic as well because they're generated with a, I'm going to say selector language, which is very similar to CSS. You basically select the span that you want to test. You say, I want this span to be equal to 200, I want this span to have a duration less than 200 milliseconds or whatever else test spec you want to add and that's it. And then based on that, if you want to integrate with any of your existing integration, I'm going to say integration testing tools. If you want to trigger from Cypress, you can do that as well, because we have this concept of a trigger where the trigger can be anything from an HTTP request, a gRPC request, a Cypress test, a Playwright test, a K6 load test.So basically anything externally that you're already using can initiate a trace test...trace-based test...as well. So whatever you're using, for whatever front end testing, UI testing, integration testing, load testing, or whatever type of testing you're using, you can add Tracetest to your integration testing. I'm going to say testing harness, and it's just going to work perfectly fine. And that's what I like about it, because the only integration points you need to care about is, okay, so Tracetest needs access to your traces, which are kept in Jaeger, Grafana, whatever you're using, and then it just needs to trigger your app. So that's the only integration point. If you have Cypress, you can trigger it with Cypress. If you have K6, you can do it with K6. So I like that modularity and that flexibility of, you can literally add it however you want and it only cares about the trace and it just works.ADRIANA: That's awesome. And the other thing worth mentioning is that trace test was integrated with the OpenTelemetry Demo recently, right? Like last fall, I want to say like just before KubeCon, North America.ADNAN: Yeah, it was a couple of months ago where I think it was...one or two of the maintainers, I can't remember. I think it was Pierre and Juliano. I actually met Juliano in person in Vienna at KCD Austria. I think it was before KubeCon in Chicago. Lovely guy, by the way. Shoutout: Juliano is awesome. Yeah, and we had a great talk about the addition of traces to the demo. It was basically him saying, oh, I literally broke the demo.I'm a maintainer, the tests passed, I merged the PR and I broke it, and I'm the maintainer. So I know how traces work. I'm very well versed in how it works. I know the ins and outs of the system, so I'm supposed to know how the system works as well. But I still managed to break it and I still managed to break it with passing tests. So that was a pretty worrying factor there. And that's when we started chatting through the GitHub issues and we said, yeah, well, I mean, let's try it, let's add it in, see how it works, add some integration tests. And then we did, and it's right now in the OpenTelemetry Demo.It's under the test directory, under Tracetest. You can check out every single service in the demo, has a dedicated set of tests and a test suite that runs basically on every PR, if I'm not mistaken. Right now, I think they did add that. Is it only on merges? But yeah, it's in the demo. If you want to take a look. What's also super cool is that as a spin off of that, we did the CNCF live, what's it called? I think cloud native Live team, we did a webinar, Whitney Lee and myself, we did a webinar just workshop showing how it works. So if you actually want to see me actually coding that, you can check that out on the CNCF Cloud Native Live podcast as well. Otherwise, just jump over to the demo. Yeah, I mean, easiest way for people to do it.ADRIANA: Yeah, I'll include that in the show notes. Yeah, I saw you posted that on LinkedIn. I was like, damn, I didn't know.ADNAN: They posted that on the landing page for OpenTelemetry. So that was quite humbling experience.ADRIANA: Yeah, that's very cool. Yeah. It's so nice to see, I think the integration of Tracetest in the OpenTelemetry Demo. And for those who aren't familiar, the OpenTelemetry Demo is based on the Google Hipster Shop. And it's basically the idea is to showcase what OpenTelemetry can do in kind of a complex-ish, sort of multi-microservice scenario written in different with microservices written in different languages. So it is not a simple application. There are a lot of moving parts and you can get it set up without too much effort, I would say locally, like using Docker Compose or if you're feeling adventurous, in Kubernetes. And so it's really cool to be able to integrate trace-based testing through Tracetest in the OpenTelemetry Demo because again, it's another piece of the puzzle, right?That's being put in, right? Really showcasing all the cool things that you can do with traces. It's not just for troubleshooting your production code, it is also for troubleshooting your development cycle, which is super exciting.ADNAN: It's normal. We've done test-driven development for years. I mean, come on, it's not a new thing because of the complexity of the systems we have now we're adding tracing to figure out what's happening. I mean, if we're already adding tracing to figure out what's happening in our development cycle, let's use that in the development cycle to also do the tests during development and then also integrate those tests in your CI, which is. I mean, it just makes sense for me. I'm not quite sure how is to put it.ADRIANA: Yeah, absolutely. I totally agree. It's one of those no-brainers, I think whenever someone comes up with a simple solution...and you know what's simple? When people are like, Oh yeah, that makes so much sense. Why didn't I think of that before? Well, there you go. Then that's when you know that this was the right thing to do, right?ADNAN: Yeah, definitely. Definitely. Yeah. The Demo is really awesome. I think it was Daniel from our team who wrote a super, I'm going to say, detailed blog post on how he actually added trace test to the demo, to the OpenTelemetry Demo. So that might also be just a reference point for people that want to get started because it's a nice reference point because if they want to actually contribute to the Demo and try it themselves, checking that blog post will actually make sure that they run the tests correctly and that any change they make, they actually won't break the Demo on merging.ADRIANA: Yeah, that's such a really great idea because I think I'm a strong believer that if you want to contribute to OpenTelemetry, you don't have to boil the ocean, you don't have to like, oh, I've got to be a contributor on the SDK or API or whatever. Something as simple as there's something in the OpenTelemetry Demo where maybe there's a feature request open. You can take that on in a language that tickles your fancy. Such a great way to get started with contributing to OpenTelemetry and now having something to make, I guess the testing a little less scary, or at least to help you understand if you break the application why that happened. I think having the trace-based test integrated in with it can be such a relief, if you will, because it's like, okay, I know where to look, I see what's going on. It's not like panic.ADNAN: You also get just system overview. When you run a test, you go, okay, so this API is going to touch all of these parts of my system, and then I actually know what it's touching. And if I know what it's touching, I know actually how to go in and either improve it, fix it, or run another test after I'm done with my development cycle. So I don't know if you're new to a project or if you're an open source contributor, you have no idea how that stuff works. You're just kind of trying to read the documentation, trying to look at the architecture, trying to figure out what API does what. And then you end up breaking something and then you don't have a test for it. To make sure that you know that it's broken is just a nightmare. Right? And then let's think about, yeah, nowadays, with remote companies, people working all over the globe, you have distributed teams, you have teams of, I don't know, eight to ten people working.How do I know what my colleague on the other side of the world is doing in another team? And whether that...it's just nuts, right? You need to have a way of having a reliable architecture overview when you're running your development. So your development cycles need to have a very nice systematic overview, and then your tests need to cover the happy paths of all of that, of the architecture that your system is. I mean, it's just freaking, I don't know, I think it's the future, more or less.ADRIANA: Yeah, absolutely. As a final plug for the power of trace-based testing, in theory, it makes sense, but to be able to actually see it in practice in a complex scenario, because when I first played around with Tracetest...I played around with Tracetest when it was barely out, I think it came out, I want to say May 2022. And I was like, let's get this to run in Nomad for fun. So I got a little example working on my own. I'm like, cool. But then my next thought was like, how does this work in a complex scenario, right? So being able to see it work in a complex scenario, I think is very opening and really shows the power of trace-based testing and what it can do for you, right?ADNAN: Yeah, exactly. And yeah, the roadmap right now is pretty extensive, so we have a lot of cool ideas that we want to start implementing. But yeah, I'm going to say the k six and the cypress things integrations regarding the triggering, those have been, for me at least, super exciting. I'm going to say most exciting just because we're getting true end to end testing. Finally, your UI test is generating traces from your browser that then triggers your backend. That's then generating traces across your entire backend. So basically you have an end-to-end test that covers your entire path of everything. And that's not something I've seen anywhere before.Right. So that's going to be really cool thing once people start using it. We just basically released it a couple of days ago. We did our announcement webinar yesterday, by the way, for the Cypress integration. Yeah, it is toasty fresh.ADRIANA: Very nice. Hot out of the oven.ADNAN: I'm just waiting to get people to start using it. I'm thinking it's going to be super cool.ADRIANA: Very nice. Well, we are coming up on time, but before we sign off, I was wondering if there's any parting words of wisdom you would like to share with our audience.ADNAN: Yeah, wisdom. Well, I'm not great on wisdom. Humor is probably...ADRIANA: Oh, how about a joke? Tell a bad joke.ADNAN: Way too grim. Let's do parting wisdom. Work out. Do stuff that you like when you're not working and work becomes easy.ADRIANA: Yes. I really like that. As a fellow workout fan, I fully support that, definitely.ADNAN: Because we're sitting around for most of the if. If you don't do something either in the morning or after work or even during your work break or whatever you want to call it in between, do crossfit, weightlifting, strength training, bouldering, biking, ice hockey...whatever you guys in Canada do. I don't know what you guys in Canada probably ice hockey. You probably skate to work. Ice skate to work.ADRIANA: Not with the weather like here in Toronto. It's been like 4C. Yeah. Yeah. I guess hockey is the thing in Canada. I am a terrible ice skater. I know, I know. I can get from A to B, and mostly stop.ADNAN: I'm a terrible skier, as you can see. One, one.ADRIANA: There you go. There you go. Oh, yeah. And you said you're a good skater.ADNAN: Yeah, I've been skating since I was, like three years old. So skating is good, but skiing totally different. You know, when you do the crossovers when you're ice skating. Because my muscle memory is ice skate. Muscle memory. With my. Tried doing it on skis. I face planted and just slid through the snow. It was not nice. It was not nice.ADRIANA: Yeah. Believe it or not, I'm a better skier than an ice skater. I went through a skiing phase, but I haven't skied for a very long time.ADNAN: Just waiting for the weekend because it's going to be cold again. So I'm just going to drive up to the mountains and do some skiing.ADRIANA: Oh, nice. See, you have snow. Toronto is like, I don't know...like, the rest of Canada is getting pummeled with snow, and we're kind of in this little island of, like, where everyone else gets snow, we get rain. So I'm like, it doesn't even feel like winter.ADNAN: Vancouver is super sad.ADRIANA: Yeah. Is it cold? Yeah, it's gotten cold right in Vancouver, which is also unusual for this time of year. For Vancouver, it's like rain. We're getting Vancouver weather. Yeah. Well, and on that fun note...Well, thank you so much, Adnan, for geeking out with me today, y'all. Don't forget to subscribe and be sure to check the show notes for additional resources and to connect with us and our guests on social media. Until next time...ADNAN: Peace out and geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:Daniela Baron is Staff Engineer at FundThrough. She has over 20 years of experience delivering software solutions for a variety of product, project and SaaS based companies with many languages and frameworks including Ruby on Rails, JavaScript (Node.js, React, Ember, Angular), Go, Python, and Java/Spring/Hibernate. Specialties include analyzing complex business requirements, writing maintainable code, implementing best practices such as linting and code coverage, engineering documentation, test automation, continuous integration/continuous deployment, and mentoring. Passionate about continuing education.Find our guest on:LinkedInDaniela's BlogFind us on:All of our social channels are on bento.me/geekingoutAll of Adriana's social channels are on bento.me/adrianamvillelaShow Links:MOOCsPluralsightTuts+Wes Bos (instructor)Erik Kennedy (instructor)GitHub GistMarkdownOpenStackHashiCorp VaultHashiCorp ConsulHashiCorp NomadHashiCorp Nomad CLINomad JobspecNomad TemplateBraintreeGatsbyTranscript:ADRIANA: Hey, y'all, welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host Adriana Villela, coming to you from Toronto, Canada. And geeking out with me today, I have Danielle Baron. Welcome, Daniela.DANIELA: Hi, thanks for having me.ADRIANA: Super excited to have you on. So where are you calling from, Daniela?DANIELA: Also Toronto, Canada.ADRIANA: Yay, Toronto. Okay, I'm going to start with some lightning round questions. First off, are you a lefty or a righty?DANIELA: Right handed.ADRIANA: All right, do you prefer iPhone or Android?DANIELA: I've actually used both. And I prefer iPhone mostly because Apple seems to send security patches for a lot longer than I've gotten on the Android phones in the past. So if you're not getting security patches, your phone is basically a very expensive paperweight. So that's why stick with iPhone.ADRIANA: Yeah, fair enough. Awesome. Next question. Mac, Linux or Windows? What's your preference?DANIELA: Yeah, that's another one that I've used all of them, and I'd say I'm happiest when I'm using a Mac. It has all the Unix utilities, a nice customizable terminal, but things just work for the most part. Like if I need to do video conferencing or watch YouTube videos, I don't need to fuss with it. So I feel like it has the best of all worlds.ADRIANA: Yeah, I totally agree. I used to have a dedicated Linux machine and then I realized that I needed to either dual boot Windows because it didn't have all the things that I needed, or I had to do a Windows VM, which in itself was like its own special nightmare. So I totally agree. Mac was like, oh, the answer to all my problems. Cool. Okay, favorite programming language?DANIELA: I'd have to say Ruby. Yeah, definitely optimizes for developer productivity and developer happiness. Yeah, I'm just in a really good mood when I use it.ADRIANA: I feel you. There's something to be said for programming the pleasure of programming. And there are some languages that bring the best out of you, and there are others that just make you angry. I found Java kind of made me angry whenever I would code in it because it's like, so verbose. And so I found my happy place with Python. So I'm glad you found your happy place with Ruby. Cool. Okay, next one. Dev or Ops?DANIELA: So for my career I've done mostly dev and I'm very happy doing that. But I have done a couple projects that were, I guess in the DevOps space and that was really cool to see that aspect of it. But mostly I've done dev.ADRIANA: Which one do you like better?DANIELA: I'd say if I could only do one, I'd stick to Dev because there's something very satisfying about building software, like working with the product team, figuring out what to build, actually building it and shipping it. There's just something very satisfying about that.ADRIANA: Yeah, totally. I totally feel you. Okay, next question. JSON or YAML?DANIELA: Ideally, I like to work in things that don't require too much configuration so you don't have to read too much of either of them.ADRIANA: Fair enough.DANIELA: Yeah, either one, really. I guess with YAML, when it gets really deeply nested, sometimes I get lost in all the white space. Which level am I at? Like if I hit the enter key and I want to go back, how many levels back do I need to go? I mean, editor support can help with that a little bit. Yeah, YAML is okay as long as it doesn't get too long and too nested. I find then it gets a little hard to read.ADRIANA: Yeah, fair enough. Fair enough. Okay, next question. There's two more left. Do you prefer to consume content through video or text?DANIELA: It really depends on what I'm trying to do. If I'm in the middle of a problem at work, like I'm getting a stack trace or I can't install something, I'm getting a weird error, I just want a text-based solution. I don't have time at that point to watch a video that's going to explain to me the root cause of the problem. I just need a quick fix. But if I'm learning, like if I've set aside...okay, I have an hour, I want to sit down and learn a new topic. I actually find video is better for that.ADRIANA: Interesting. I like that. Cool. Okay, final question. What is your superpower?DANIELA: Oh my. I guess I'm strong on the written communication skills. Whenever I figure out a problem, I like to write either a blog post or if it's something very specific to work that I'm doing for my employer, I'll write up some internal documentation and include that on my next PR. Basically, then next time that problem comes up, if I've forgotten or I'm not around, then other people can fix that. So yeah, I'd say definitely good written documentation skills.ADRIANA: Yeah, and I can totally vouch for that because you write some really great blog posts and they're fun to read. They're super informative. So yeah, I definitely appreciate that. And I think having very strong communication skills is something that isn't the first thing that comes to mind when we talk about folks in tech. And yet it's such a crucial skill, right? Because yes, we communicate through code, but we must also communicate through whatever the official workplace language is so that we can understand each other and so that we can build better software together.DANIELA: Definitely, yeah.ADRIANA: Cool. Okay, well, now that we're done the lightning round questions, it's time to get into the meaty bits. So one of the things that you and I talked about before getting on here was something that's so key to developers, which is learning new skills. So I wanted to just get your thoughts around that.DANIELA: Yeah, definitely. Throughout my career, I've always had to learn new things. My educational background is computer science from university, which, at least the program I took was very theoretical, which meant that when I got into the real world, I didn't actually know anything, practically speaking. So yeah, I've always had to be learning. So early in my career, I used a combination of just try to get things working and then look it up. If it's not working, you get an error. Look it up on StackOverflow or other free online content like blog posts and tutorials, and you can definitely make some progress like that. But I always had a sense of that.I was just reacting to each problem as it came up and I didn't have a sense of big picture understanding. And I felt like, well, if I understood the kind of bigger picture of how this framework or language or whatever works, maybe I could plan things out more smoothly so I wouldn't be running into these little problems here and there. Yeah. Then I've done some formal education. This is like in-classroom kind of courses where there'll be a workshop, like a two-day or sometimes it's a whole week, and if you're lucky, your employer will actually pay for you to go and they'll send a few people from the same team to go learn together. And it's nice to learn with your team and to have a complete change of scenery so you're not at your desk trying to learn while you're also trying to put out fires or something. You actually have a whole day or a whole week to focus. So that's really nice.There are some drawbacks, so there's not much flexibility. Like, if someone needs to leave early to pick up their kid or has an appointment or something like that. Like these courses that they're in person, they tend to be like, let's say from nine to five. And if you have to leave for something, that's too bad, you're just going to miss out on the content. The other thing I found with some of these kind of in-classroom, in-person sort of courses, the pacing may not be appropriate for everyone. So sometimes what will happen is there'll be a section that maybe you already know it, it's kind of introductory, and you've already self-taught on that and you kind of wish the instructor would speed up through that, but they can't because there's other people in the class. They might not know that and it wouldn't be fair to them. On the other hand, sometimes there's some material that's really tricky and you might want to pause. I've felt like that in classes where I wish I could just pause the instructor and go explore a little bit on my own or maybe try it out and then come back.And you can't do that because the instructor has an agenda. They need to cover it because that's what everyone's paid for. So yeah, that's some pros and cons of that. I've also done MOOCs. So for those in the audience who may not have heard this term, it stands for massive open online course. And it's a kind of online learning platform that offers either free or low-cost courses, typically from universities. So it's a lot of kind of university topics and it's accessible to a global audience. So in theory, there's no limit to the number of participants you could be learning with people from all around the world.Yeah, it's really cool. My experience with these has been the quality of the material is really high. It is like university quality material. Usually there's like a series of video lectures that are released each week, but you do it from home, so it's more convenient. Like, you don't need to commute and be at a class at a particular time. You can watch the videos whenever you want, but there's always a drawback, so there's no accountability. No one's taking attendance or cares. Whether you're actually watching the video lectures or not, it's totally up to you.So you need a lot of internal motivation to get through the material. The other thing to watch out for is some of them do have assignments that need to be handed in each week. So although you could watch the videos whenever you want, if you don't keep up with it, it just tends to pile up. So you do need to set aside some regular time where you're going to do the lectures and do the assignments. And I actually found the workload was surprisingly high, like 6 to 10 hours a week that you need to succeed at these.ADRIANA: And that includes the assignments too?DANIELA: Yeah, like watching the videos, understanding the material, and completing the assignments. So I would say for anyone who's considering these, maybe evaluate what a typical week looks like in your life and see if you think you have six spare hours. And if not, I would urge caution before signing up because it might just create more stress in your life.ADRIANA: Are these like paid things as well? Like, these are paid courses?DANIELA: Some of them are. So when I took...I took some at Coursera and it was free. I think the way they have it right now is it's free if you're just going to watch it. But if you want someone to grade your assignments and you want to get some kind of certificate of completion or something like that, then there's a paid version. I don't have the prices offhand, but it is significantly less than what you'd pay for like a four year degree program at a university, something like that. So it is still a good choice for some people, but you definitely need the time.ADRIANA: Yeah. See, this is why, honestly, hats off to people who do school part time. Get some other degree while they're working. Because the thought of doing that, just like, I'm so over school right now, I do not want to touch that. It's been 20 plus years since I finished school, and the thought of trying to juggle that just does not tickle my fancy. I'd much rather sit off in the corner and be like, oh, this is a cool topic that I want to learn about. Let me just read up on it for me. Anyway, I know everyone's got their own style, but yeah, I think that would turn me off from that kind of thing. It's a little too structured for my taste.DANIELA: Yeah, very structured. And the time commitment is pretty big. So actually that leads me to the next kind of learning, which actually you might like better. So it's these online screencasts, so it's sites like Tuts+, Pluralsight. There's some individual instructors that offer these like Wes Bos and Erik Kennedy have these kind of longer video courses, but they're usually completely self-paced. And what they do is the video sections are split up into very short sections and they tend to be pretty practical and hands-on. Sometimes videos are split up as short as just ten minutes. So if you're the kind of person that doesn't have a lot of time, but maybe you could squeeze in 10-15 minutes a few times a week.The online screencast might be a really good choice for you. So you can watch a video and then try out some hands-on exercises, like follow along with the instructor. Because it's a video format, you can pause it, you can go explore other areas, like if there's something you don't understand and you realize, oh, I want to do a little bit more research here or try this out and then I'll come back and finish this video. It's totally up to you. It's self-paced, so the flexibility is like maximum flexibility on these. But again, you need a lot of internal motivation because there's no assignments to hand in, no one's checking up on you. These do tend to be paid services, like Pluralsight, for example. I think they have either a monthly or an annual subscription. So you might look after a year and say, oh well, I only use this for like half an hour, maybe I shouldn't renew my subscription.ADRIANA: Yeah, totally.DANIELA: But that's pretty much the only accountability there is. If you care about the money.ADRIANA: The incentive of like, "I'm paying for this, I should get the most out of it."DANIELA: Yeah, but that's definitely one of my favorite ways to learn. I found it very effective for learning all kinds of Javascript libraries and CSS frameworks and things like that. Another way to learn that I've used is with books, like programming books or technical books. This has some of the same qualities of the online screencast in terms of it's totally self-paced, ultimate flexibility. But some people prefer to learn in reading rather than watching videos content. And I find books do tend to go more in-depth than the videos. So this might be good for some people. Just a hot tip, definitely invest in a bookstand.I got one from Amazon for like $20 and it allows you to prop up the book right next to your monitor at a natural angle so that way you can keep your head and your back more straight when you're reading rather than being kind of hunched over your desk. Like if you have the book laid flat on your desk and that's going to make such a difference in terms of the ergonomics. So yeah, that's books. And finally, after some 20 years of experimenting with different kinds of learning, what I found works, I think the ultimate for me, is kind of using a combination. So if I'm starting something that's brand-new to me and that I need to learn, I like to take kind of an intro level screencast course just so I can understand the nomenclature. Like each kind of tool that you're going to use has different terminology. And if you start trying to Google and you don't even understand the terms, none of it's going to make sense to you.So I like to take an intro level course just so I understand the very basics, and then I like to actually use it on the job. And there is where I'm going to encounter problems that are more complicated and couldn't have been covered through an intro level course. Then I use a mix of looking up online like StackOverflow blog posts, AI, LLMs, newer things, ChatGPT or whatever, and then I can understand a little bit better the answers to those, or I even know what questions to ask because of the basic course I took. And then if I find I'm still using that at work a lot, I might circle back and then either get a book or take a more advanced level course to get a deeper understanding. What I found for me is if I take a really in-depth course right from the beginning, before I've even tried, I just won't appreciate the nuances because I haven't encountered those problems yet. So I think doing kind of an iteration of some simple introductory material and then some hands-on, try to solve real problems with it and then come back for more learning. If you feel like you still want more in-depth, that's kind of sweet spot for me with learning.ADRIANA: That's awesome. Yeah, that's really great advice and I definitely appreciate you going through all the different options for learning things because I think ultimately learning is such a personal thing, but knowing what options you have out there as a learner so that you can get started, because sometimes it can be very overwhelming, right?You want to learn the thing but you don't know how, and then you have a very particular learning style. So knowing what's out there, that's going to suit your learning style and then playing around with it as well, because it might not be, as you said, one thing that is the answer to all your problems, right? It could be a combination of things, I think is super important. I wanted to go back to a point that you mentioned earlier, which I thought was very interesting where you talked about reactive learning. And I think in our field there is a lot of reactive learning because either it's because you're thrust into a project on X and all of a sudden you're like, oh, I know nothing about X, I must learn this. Or like you are doing something that you know how to do, but then you start refining your code and digging a little bit deeper and you want to make it a bit prettier and then all of a sudden you're like, "Oh, I actually don't have the skills to do this. I must learn." And I think that the reactive learning can be kind of exhilarating sometimes, but also very stressful. I was wondering if you could share your thoughts around that. How do you deal with reactive learning, especially when you're under the gun, you've got a deadline and you got to figure out this thing.DANIELA: Yeah. One thing, it helps if you're working in an environment where you have psychological safety, meaning if you can say to your manager something like, look, I haven't done this before, but I'm happy to learn. So I might make a mistake or it's going to take me a little longer than someone who has five years experience in doing this thing, and I'll document as I'm learning. So it will help the next people that have to do this. So, yeah, it definitely helps if you're in an environment where you have the safety to do that because otherwise it is very stressful.ADRIANA: Yeah, I completely agree. I mean, there's nothing worse than working for a manager that's breathing down your neck and then you're feeling this extra pressure to learn, and then all of a sudden something that could have been fun and exciting turns into this complete nightmare scenario and almost seizes up your learning. And that doesn't do anyone any favors. So, yeah, I completely agree. The psychological safety is so important, and it's a recurring theme in tech. I mean, we have to have psychological safety within our teams, right? So that we can be our best selves when we're at work, right? So that we can be as productive as possible, so that we can learn as well as possible and so that we're happy because ultimately we're at work for a big chunk of the day. So it had better be like a relatively happy place, I would hope, right?DANIELA: Yeah, I agree.ADRIANA: Cool. Then the other thing that you brought up, which I was getting, like, flashbacks, the 9-5 courses, which, as you said, it's a great way to take yourself out of the day to day and attend these short little courses with your coworkers. But as you said, everyone's kind of at a different pace. And then when you hit the point when the instructor is talking about something that hasn't sunk in yet, and you're like, can I just hit the pause and rewind button? And they often don't have time for you. And I found that it even brings me right back to my university days being in lectures where I'm like, oh, my God, I am so lost. And then you're asking the professor questions, and after a while this happened to me. After a while I had this one professor, he's like, I'm not taking any more questions. I have to move on. And I'm like, no, I'm lost. Help. It can be so devastating. What do you do in those types of situations when you're trying to keep up, but sometimes the material, like you hit a snag in the material and the comprehension.DANIELA: Yeah. So I haven't done any of those courses like that recently, and that's because of exactly the problem you're describing. So what would happen to me is I would just ask, okay, are we going to get this material? Are we going to get access to the slides or whatever and just hope that I can have time to review it later? There's not a whole lot you can do if the instructor is clearly in a hurry to cover more material. What I learned from experiences with that kind of training is that it's just not the best use of money for me. And those courses can be expensive because once I hit a point where I don't understand it, the rest of the material builds on that. So I won't understand the following either. And that's really why screencasts and books are my preferred. Probably in the last five to eight years or so, I've just been using mostly screencasts and a little bit of books because I find I get way more out of that, like just the ability to pause the instructor anytime I need to.And it will take me a long time to get through. There could be a 1 hour video course, but it could take me weeks to get through it because I'm just doing a little bit at a time and I'm actually hands-on trying the exercises. And if I get an error, then I'll go look that up and understand, okay, what is it I did wrong? Or if there's a certain API they're using, I'll wonder, oh, what are the other flags I can pass to this? What's the other behavior? So I'll do more exploration that there would be no time for in a formal course, like an in classroom kind of course, and then I'll take the time to organize my notes about it. I might publish my notes to GitHub so I can always find them, whatever computer I'm on. Yeah, basically that's why I find the self-paced learning more effective for me because a) I have a high degree of internal motivation, so some people might need the in classroom setting to like, otherwise they'll just never get it done. But that's not the case for me. And just the ability to pause the content is super valuable.ADRIANA: Yeah, I totally agree. My mind tends to wander. And so if it wanders at the wrong time where the instructor is talking about something crucial and it's a live course, you are screwed. The other thing too, about the live courses is I always find like, they're great for intro materials, and so sometimes you'll come out of it thinking, oh, I get this. And then you go to apply this at work and you're like, oh my God, this was like the simplest use case ever. And of course the use case at work is seldom ever the simplest use case. It's probably like the most difficult, weird ass use case ever. And so it almost feels like it doesn't get into enough depth so that you have the high level understanding.But do you really understand it? Because it doesn't give you necessarily the tools you need to get into those more complex use cases. And then the other thing that you mentioned that I thought was really interesting, which I started doing myself, is the idea of publishing your own notes to GitHub. Because I used to keep a bunch of notes in the text editor on the Mac or whatever notepad on Windows, and my notes were always so freaking messy. And I'm like, man, it would be so nice if I could put these in Markdown and make them searchable. And then I'm like, "Wait...GitHub! What?" So creating Gists or GitHub repos of notes for me when I realized, "Oh, I can do that!" was super useful.DANIELA: Yeah, definitely. It helps you to organize. And Markdown is a very lightweight format, so you don't have to fuss with the WYSIWYG editor and all the bugs that can sometimes be in that markdown is just so simple that it really gets out of your way and just lets you focus on the content that you want to type up. And yeah, I like publishing them. Sometimes I get stars on them from people I don't know or whatever. So I think they definitely get indexed and show up in search results. So maybe if my notes can help out some other people too, then that's great.ADRIANA: Yeah, totally. That's always been my philosophy too. Whenever I learn something, I'm like, I can't be the only person who has struggled with this because I don't know about you, but I always tell people, the thing I hate the most about technical documentation is how it feels like they started out explaining the thing and then partway through the author is like, "This is too much work, I can't deal with it. You figure it out." It's like the thing in the math books. We'll leave the proof, proving the proof up to the reader. And it's like, "No, show me. I don't know how."DANIELA: Yeah, there's definitely value in seeing an example as well. Sometimes with the official docs, they're just showing you one line or they're explaining all the options that you can pass to method call. But if you see it in a working example with explanations of like, okay, this option is changing the behavior because of this. And here's the output from this that can be a lot more helpful to people. So I'll write my notes kind of in that format.ADRIANA: Yeah, I completely agree. And one of my biggest pet peeves with documentation, just building on what you were saying around these sites will show you little snippets of configs. And for me, one of the things that annoys me, one site that always gets me is the HashiCorp site. So, HashiCorp people, if you are listening: pro tip on fixing your docs, include a full example of your stuff. Because having, like I was saying one time I was going through the docs, I thought it was like some configurations for a Nomad job, but it turned out that it was some configuration that applied to the actual Nomad agent configuration. And I'm like, could you have been a little bit clearer? So I think a full-fledged example would have been super helpful. It's the same sort of thing for me. Whenever I write blog posts, I like to have the full example because I always think of like, what if it was me reading this and I'm learning a thing from scratch and I have no idea what's going on? Give me the example. Give me links to the things that you're talking about. That might not be super obvious that you obviously knew about, but me, as a beginner, I have no freaking clue what's going on.DANIELA: Yeah.ADRIANA: Cool. Actually, speaking of Nomad, because I wanted to get your thoughts not on Nomad itself, but you and I both worked at a place where Nomad was the orchestrator, the container orchestrator of choice. And we both found ourselves at this workplace in a position of like, well, I've never used this before, so I was wondering if you could talk about what your process was around learning how to use this tool that you'd never touched before, may not have heard of, or maybe heard of in passing, to be able to ultimately do your job well, sure, yeah.DANIELA: Well, just because I've been doing this job for a long time. In the old days, developers didn't even have access to deploy, especially to production. This used to be really tightly controlled by gatekeepers, like an operations team or change management teams, if anyone remembers things like that. And I wouldn't want to go back to those days for sure. But it is ideal if deployments can be made easy, so that not everyone needs to invest time to become an expert in it. Ideally it should just work with like, oh, you just get merged, your branch to the mainline and it should just deploy. But it is good to have some understanding of how it works. So yeah, I had an experience in one role I was at.We were going through a platform migration. I was a developer working on a Rails application at this job, and we had a MySQL database, a background task runner, some other background services, and a number of cron jobs. And the way deployments were done, it was with a mix of like Zen and some homegrown tooling. There was a JIRA-based change management approach that required manual approvals. And then there was Jenkins. And it took a long time to get a build together. Like a developer had to spend time doing it, and you could only do it on certain days of the week. And the idea was to change everything, to have all the services running on a private cloud built with OpenStack and using HashiCorp tooling.That's what this company had decided to go with, including Terraform, Consul, Vault, and Nomad. And the other goal was to have everything, all the deployments be automated with GitHub Actions. So we were already using GitHub. So it was kind of a natural fit to use GitHub Actions rather than go some third-party CI. So originally the task of doing this platform migration had been assigned to another developer on the team. But shortly after this project started, it turned out actually he was leaving. So my manager asked me if I wanted to take this over. Now, I had not previously done anything like this.I worked with build systems and deployment system, but I'd never set it all up from scratch and kind of defined how it should work. So I was a little nervous, but also kind of excited for the opportunity to learn something new. And that psychological safety I was talking about earlier was really high at this workplace. So I felt really comfortable saying like, yeah, I'd love to learn this. I don't know it now, but I will. What I did to start with, because we were using Nomad, I did take there was this introductory kind of video course on Pluralsight, so I took that it wasn't really so much hands-on, it was more just explaining the concept. So I was kind of able just to write down the definitions of everything at this point, since I hadn't worked with it, I didn't really understand, but at least the words like jobs and scheduling and resources and services and tasks, these kind of terms that are going to come up as soon as you start trying to figure out nomad stuff, at least they sort of became part of my then. So I had to learn about the concepts of Nomad.Like, you configure things with HCL, which is a HashiCorp configuration language. It's kind of their own language, but it looks close enough to a mix of YAML and JSON that I was like, okay, it's a little different, but not too different. Yeah, I see what's going on here. And I had to learn about the jobspec file and how you, that's what HashiCorp uses to configure a job that needs to be scheduled, where scheduling means run a task. Like I have a Rails server that I want to run, so that has to be in a task. And I had to learn about lifecycle methods because it's like, well, before you run the Rails server, there might be database migrations. So how do you run those before. Oh, they have lifecycle hooks.Yes, please run the database migrations before you run the Rails servers. And I had to learn about how you can nest groups and tasks inside the jobs. And we had decided to use Docker. So I had to containerize our applications and then learn how to use the Docker driver and how to configure that, how to tell it, which command to run and how you can also specify resources like cpu and memory usage. So how do you do that? How do you specify health checks? One thing with Nomad is to get resiliency, like auto-restarting failed jobs or doing rolling updates. You can do that. They provide a number of stanzas to do this. And the stanzas are like sections or pieces of configuration that you can define in the job.They have a bunch of them, like update, restart, check, restart, reschedule, migrate. So I had to read up on all of those. And they do have pretty good docs, like at least defining all those terms.ADRIANA: Yeah, totally.DANIELA: And then a process I would run into is I would put what I thought was a fine jobspec together and then I would read up on the Nomad CLI. So, oh, you install it on your laptop and then you do like Nomad job, run or run job. I can't remember the exact syntax right now. And then it submits that job and then you can check on the Nomad, there's a web UI and you can check. And I would frequently get errors. And this is where things got a little dicey. Sometimes I would Google those error messages. This was before the days of LLM.So yeah, you just had to Google or DuckDuckGo the error messages and sometimes nothing would turn up. So another technique I guess for learning or troubleshooting is that if the project's open source and Nomad is, it's written in Go and it's on GitHub. So you can go to the GitHub source and search that error message in their code, find out where from their code it's coming from. And then if, you know, I had used go a little bit before, so I know just enough that I can kind of trace through the code and see, oh, okay, maybe this is the problem, then come back to my jobspec, try something else. So yeah, it was very iterative. I also had to learn how to use Vault. That's another HashiCorp product, and Vault and Nomad talk to each other. So that Vault is for secrets management.And what we needed to do, since we had decided to store our secrets in Vault, you need to generate those as environment variables in our case for the Rails application, like what's the database host and password and other services that we needed like Braintree, API key and all those things were in Vault. So this was another kind of tricky part of Nomad is there's a template stanza, and you can use that to extract things from vault and then generate an Env file and make that available to the container that Nomad is going to be running as your task. And then all those things become available as environment variables to your application. So I had to learn how to do that and learn about the Nomad CLI. Some more learning I had to do was about GitHub Actions. Fortunately, that's pretty well documented as well because once you know how to say, run a nomad task from your laptop, you're like, okay, well, I don't want to have to do this each time there's new code. So I had to automate the process of, okay, so if a PR just got merged and all the tests are passing, then what we want to do is build a Docker container and tag it with the latest Git commit SHA and then make that available through variables to Nomad so Nomad will know, oh, this is the container that I need to pull and run. So putting all those pieces together, one thing that helped me with the GitHub Actions is I actually did a little experiment on my own and built a CI/CD pipeline for my blog, which is a static site built with Gatsby and has some Rails service as a backend.It's not nearly as complex as what I had to do at work, but I found it really helpful to kind of experiment in a low risk way, like just on my own as a side project. And then I actually ended up writing a blog post about how to do that. And then I was able to take some of those things that I had learned about working with GitHub Actions and build our real CI CD pipeline, like for big project for my employer. So yeah, that was really good project. It was kind of the most DevOps I'd done up to that point in my career. And just seeing how everything works under the hood, I set it up with as much automation as possible. I forgot to mention, another kind of automation you would want is sometimes developers want to just deploy a branch to like, we had other environments beside production. We have dev environment, staging. So I set it up that you could make an empty commit on your branch with a special keyword like deploy dev or deploy staging.And with GitHub Actions there is a way you can read the head commit message and say okay, if it's not the mainline, like this is a feature branch, you can get the branch name and you can check that get commit message and say oh, okay, I'm going to deploy this to dev or deploy this to staging. And you use that together with something called environments, which is another feature that GitHub provides. And if you do that, you actually get slack integration for free. So it will post a message to a dedicated Slack channel saying oh, deployment to dev starting or to production. And then if it succeeds or it fails, so you get status information like that without having to monitor nomad directly yourself.ADRIANA: That's awesome.DANIELA: Yeah, so that was the project. A lot of learning, a lot of automation. It was nice because being a developer, I was kind of making it for myself. I knew how tedious the existing deployment process was and I was like, okay, I don't want to do any manual work. I want this to just work and be automated.ADRIANA: Yeah, and I think that's so good because you're basically taking advantage of your knowledge as a developer and saying, okay, well, if I could make this the most optimized thing possible, this is what I would do, which honestly, that's kind of what made me fall in love with DevOps because I'm like, oh my God, why does all this crap have to be manual, right? These are the things I would like to do to make my life easier. So very cool. Awesome. Well, we are coming up on time. I can't believe how quickly the time has passed. There's just so much to talk about in this space of learning and I could be talking and talking and talking about this stuff forever. Before we part ways, is there any piece of advice that you would like to give to our audience as far as picking up a new skill, especially in tech?DANIELA: Yeah, definitely. Try to always be learning and there's so many different ways to do it. As we covered earlier, what I would...advice I would have for people is, just try different ways. If you don't like something, let's say you took one of these MOOCs and that didn't work for you. That's fine. You still learn something. You learn that that style doesn't work for you and try something else. Just keep trying different ways and you're going to find a certain style of learning that works best for you. So keep experimenting and definitely keep learning.ADRIANA: That's awesome. I really like that advice. I think it's really important for us to be better learners is to understand what works for us as learners. So awesome. Thank you so much, Daniela, for geeking out with me today. Now, y'all don't forget to subscribe and be sure to check the show notes for additional resources and to connect with us and our guests on social media. Until next time...DANIELA: Peace out and geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:As a founding member of Shopify's incident response program for nearly 7 years, Ashley Sawatsky led incident communications and processes. Currently, as Senior Incident Response Advocate at Rootly, she consults with tech giants like Canva, Cisco, NVIDIA, and more on incident response strategies.Find our guest on:X (Twitter)LinkedInFind us on:All of our social channels are on bento.me/geekingoutAll of Adriana's social channels are on bento.me/adrianamvillelaShow Links:ShopifyRootlyWinows 98Ruby on RailsSite Reliability Engineering (Book)Disney InteractiveWorking Effectively With Executives During an Incident (blog post)Black Friday (shopping)Cyber MondayAlan Leinwand (former Shopify CTO, current Webflow CTO)WebflowJJ Tang (Rootly CEO & co-founder)Quentin Rousseau (Rootly co-founder)Additional Links:Lessons in Incident Response I Learned While Waiting Tables (blog post)But It’s Not Our Fault! When Third-party Incidents Affect Your Service (blog post)Transcript:ADRIANA: Hey, y'all, welcome to Geeking Out. The podcast about all geeky aspects of software delivery DevOps Observability, reliability and everything in between. I'm your host Adriana Villela, coming to you from Toronto, Canada. And geeking out with me today is Ashley Swatsky of Rootly. Welcome, Ashley.ASHLEY: Hi. Thank you.ADRIANA: And where are you calling in from today?ASHLEY: I am in very snowy Ottawa.ADRIANA: Yay. I feel your pain. I went to high school in Ottawa, so I remember having to shovel my roof one year. Well, not me. My parents hired someone to shovel the roof. Yeah, there is a lot of snow in Ottawa.ASHLEY: Yep. After this call, I will be trudging through the snow to pick my six-year-old up from school. And it's a daily battle in the winter.ADRIANA: Oh, yeah, yeah. Between that and the freezing rain. I remember lots of freezing rain in Ottawa. I'm like, really?ASHLEY: Lots of that too.ADRIANA: Yeah. No snow in Toronto. The temperature is just like a little bit above zero. So it's just we get rain, it's like why?ASHLEY: It's a little sad. I know. Well, I'm excited to come tomorrow and get a little break from the snow, so that'll be nice. I'm only spending. I'm coming. We're going to have dinner and then I'm going to leave in the morning.ADRIANA: But it'll be a worthwhile trip.ASHLEY: It will, yeah. Lots of good folks at that dinner, you included. Can't wait. Yeah.ADRIANA: Excited, excited. All right, well, we are going to start off with, first off, some lightning round questions. Are you ready?ASHLEY: Okay, I think so.ADRIANA: Question number one, are you a lefty or a righty?ASHLEY: I'm mostly a righty, but sort of ambidextrous in some things, like golf. Oh, cool.ADRIANA: Yeah, I'm a lefty. And so anyone trying to teach me sports that require dominant hand throws people off.ASHLEY: I think it's because my mom's left handed. So it's like everything my mom taught me how to do, I do left handed.ADRIANA: That is so cool! I kind of impose my left-handedness at home with the way that I put things on hangers because I do it. Like, people who are right-handed probably don't know this unless they live with the left-handed person, which is like the way in which you orient your jackets when you hang them on a hanger. So yes, I feel you. I am that left-handed person. At least in my house growing up, my mom was left-handed as well. So there were two of us, two against two against the righties. So it was evenly matched.ASHLEY: I wonder if it's genetic.ADRIANA: I think it is.ASHLEY: Interesting. Yeah. Nice.ADRIANA: Yeah. All right, next question. IPhone or Android?ASHLEY: iPhone. Die hard iPhone. I can't do the green bubbles. Sorry.ADRIANA: I know the green bubbles make me a little bit sad. This is why I use Signal or WhatsApp rather than the messages app for non iPhone people.ASHLEY: We have a joke at Rootly because we love to have a group text going that green bubbles are immediately out. It's totally joking. We absolutely do not screen candidates based on green bubbles. But yeah, it takes some getting used to where I'm a big iPhone user and just Apple in general. As you can tell, I got the AirPods, the whole thing.ADRIANA: Yeah, I'm a definite ecosystem convert as well. If Apple had a fancy podcasting mic, I would buy it.ASHLEY: I actually checked if Apple had a mic when I bought the blue Yeti, but I have. It's just the default.ADRIANA: So sad. Okay, next question. I think I know your answer. Do you prefer Mac, Linux or Windows?ASHLEY: Yeah, I'm a Mac user. Windows would be a second. Was it KubeCon? One of the conferences we did recently? I'm pretty sure it was our KubeCon merch. I did a Windows 98-inspired sticker sheet that I made with our designer Jerry, and that was our little homage to Windows 98.ADRIANA: I remember this.ASHLEY: It was a good one. It was a good one.ADRIANA: It was like very nostalgic. I saw it and immediately I'm like, yes, it's got the vintage vibes.ASHLEY: That was like the first operating system where you could customize things a little bit, at least that I knew of. Like you could change the top of your windows to have that little gradient bar. And to me that was just like the most exciting thing ever.ADRIANA: I feel you. I do enjoy some nice customizations. Okay, next question. Favorite programming language.ASHLEY: I have a lot of fun with CSS, but I'm going to give a shout out to Ruby. We're a Rails shop at Rootly, and before I worked at Rootley, I worked at Shopify, a massive Ruby monolith. And I just got to know the Ruby community really well. And I think that the community around Ruby is unmatched.ADRIANA: It is a very vibrant community. Absolutely.ASHLEY: It's incredible. Yeah.ADRIANA: I've never known someone who's written code in Ruby to say, "This sucks." Everybody loves, loves, loves Ruby.ASHLEY: Yeah, it's kind of a love hate. I think some people, when they're new to it, they hate it. But the people who have been programming in Ruby for a long time, if you love Ruby, you will never take a job that's not coding in Ruby.ADRIANA: Yeah, I have a friend like that actually. We did some Java dev back in the day and now she's like a Ruby Rails developer and she doesn't want anything else. Awesome. I think that's great. It just speaks to the power of the language. Right? Okay, next question. Dev or Ops?ASHLEY: This feels like a trick question. I'm a technically DevRel, so I feel like I need to say Dev. But I'm going to say Ops. I thrive on the ops side, so I'm going to say Ops.ADRIANA: All right, next one. Also not a trick question. JSON or YAML?ASHLEY: Oh God. You know what? We actually had a really crazy incident at Shopify that stemmed from YAML parsing, so I'm going to pick JSON just because I'm still traumatized. I still think about that incident in writing that post-mortem.ADRIANA: It's the traumas that shape our lives.ASHLEY: It was harrowing.ADRIANA: Okay, another question that is slightly...more than slightly controversial. Spaces or tabs?ASHLEY: Tabs.ADRIANA: All right, two more questions. Do you prefer to consume content through video or text?ASHLEY: I'm old school. I like reading text.ADRIANA: Yeah, same. Yeah, give me a video and I'll probably not read it. I mean, watch it.ASHLEY: I get distracted. Yeah, I watch it and then I open another tab and then I'm responding to emails and I'm like, wait, what was happening?ADRIANA: Yeah, exactly.ASHLEY: But if you're reading, you're reading.ADRIANA: Yeah, and then if you get distracted, you just scroll back up.ASHLEY: Yeah, exactly. I would even go a step further and say, ideally, print. I actually just ordered a print copy of the Site Reliability book, like the one, the Google one that Jenn and Niall did and all the people who contributed. And I've been working my way through the actual print copy of it. It's nice. I can highlight it with a real highlighter. It's so good.ADRIANA: Yeah, there's something very nice about that. Sort of very interactive, tactile aspect of having a print book.ASHLEY: Yeah.ADRIANA: My only thing with print books, I like them, but most of my books now are ebooks because I simply do not have the room in my house.ASHLEY: Yeah, that is becoming a problem for us. Our bookshelf is overflowing. We probably need another one. But I also like it because your computer, your phone, everything, you have so many apps, so it's very easy to get a notification. And then I'm distracted and I'm responding to emails and I'm like, wait, I was trying to read something like, what happened to my attention span?ADRIANA: Yeah, totally.ASHLEY: So I try to, if I'm going to read, set a 25 minutes timer or whatever and just actually pick up a book because it's the best way for me to not get distracted because I have a short attention span.ADRIANA: Yeah, I feel you. I suffer also from a short attention span. For me, what has worked...not as nice as the tactile feeling of a book...but having a Kindle where that is all it does. So I have that with my breakfast. I'll have my Kindle book on me and just chill for 20 minutes before the day starts. And it's awesome.ASHLEY: Yeah, I should get a Kindle. That's honestly a great idea. I felt like I didn't need one because I have an iPad. But then you have the same problem. It's just like a big iPhone.ADRIANA: Exactly, yeah, yeah, I know. I remember when I was actually looking for a Kindle initially, and someone's like, just get an iPad. I'm like, no, I cannot have the distractions. I just want the one thing.ASHLEY: See, this is the hold that Apple has on me. It's a problem.ADRIANA: Feel you. I feel you. Okay, final question. What is your superpower?ASHLEY: My superpower? Oh, I like that question. I think my superpower is that I'm very scrappy. Everything I know and everything that I've done in my career has been pretty unconventional. I do not have a traditional computer science degree. Everything I've learned, I've just kind of learned through watching and doing and figuring it out and asking questions. So, yeah, I think that has been helpful for me in my life, and that's something I will continue doing.ADRIANA: I love that. I think scrappiness is very important, especially in our industry.ASHLEY: Yeah, I think it also just builds a lot of confidence once you realize you can actually just figure this stuff out. I think for a long time, it felt like there was some secret trove of information that people had that I didn't or these invisible barriers that existed. And then at some point, you realize you can figure it out. There's no secret. Everyone's literally just figuring it out also.ADRIANA: Yeah, exactly. I think that's the most comforting thing, is realizing that you're not in this alone. Like, chances are other people are just wading their way through the plethora of information and trying to sort this out. And then if we let each other know that we're all kind of trying to figure this stuff out together, then we can provide each other support.ASHLEY: Totally. Yeah. Once I got lucky to work at some very well known companies, I worked at Disney and Shopify, and I got to work closely with people I really admired in those jobs and our execs and leaders. And once you realize, of course, they've got so much experience behind them, and that's what gives them the ability to figure out the next really difficult challenge. But they're still figuring it out. They are sometimes unsure of what to do. And once I saw that up close, I was like, oh, my God, I never would have thought that these people are also just kind of making the best decision that they can and hoping it works and adjusting as they go.ADRIANA: That's true. Yeah. It's like the, we're all human at the end of the day.ASHLEY: Yeah. And especially tech. It changes so much. No one knows exactly what's coming next and what's going to work. You just have to be willing to try things.ADRIANA: I guess that's true. Yeah. Expert one day and newbie the next, right? Pretty much. So I think this is a good springboard into our main discussion. So you mentioned that your superpower is being scrappy and that you don't come from a traditional comp sci background. What is your background?ASHLEY: I'm like, it depends how far back you want to start. I think I'll start at Disney because it was my first tech job. So I worked in a part of Disney that tragically is now defunct, called Disney Interactive. And it was like the tech products division of Disney Consumer products. So it was like apps, online gaming, websites, and digital products. And I started out on the tech support side and eventually kind of like, fell into this weird role that was communications focused, that was handling special cases, basically things that had happened where somebody's experience with Disney was not optimal. They've gone through the support team, they have not gotten the resolution they need, and they've filed a case. And I would deal with those cases and talk to people and try to just make it better, make it magical.And I had a lot of fun doing that. And that was like one of the first times, I think, in my professional life that I had discovered something I felt like I just kind of had a knack for. So I was spending a lot of my time fixing things that were broken, but they were more experiences, they're more technical things, too. They bought a game and it's not working, and we've got to file a bug report, but it spanned a lot of different things. And so that was when I was, ah, I kind of like having hard conversations and solving problems, and it was probably one of the first times I had really felt challenged in a role, too. And I did that for some time. And then a friend of mine - shoutout to Colin, who I worked with at Disney - he had recently left and he had joined this company called Shopify. And he was like, this is so cool. You need to join Shopify. It's so much fun. I was like, it sounds like a weird multilevel marketing scheme. Like, what are you talking about? This was in 2015. He was like, you get to work from home. They send you a laptop. And I was like, that's a scam. Turns out now Shopify is very well known and is, in fact, not a scam. It is a very real global software company. So all that to say, I joined Shopify and I was a founding member of this incident response team that they were building out. And this was very early on in incident response at Shopify. So it didn't actually have much of a technical focus. It was a bit more focused on the customer support side of things, where, again, something had gone very wrong and we were reacting to it, trying to make it right. If something had gotten to the three of us that existed in incident response, it was because they had gone to an executive or had gone public with some issue, or we made a big mistake and it needed fixing. And so as we were building that out, we started realizing that a lot of the solutions for this problem should be solved at the support level, because you can't have three people responsible for everything that goes wrong.It didn't make sense for this to be an escalation to a separate team. We thought anybody should be able to handle something like this, a customer issue, and fix it. So we focused a lot on that enablement side, like how to have hard conversations. What are the rules? When do we give a refund? None of that was defined. And then, as we had sort of to say, worked ourselves out of that job of being the escalated customer support, we ended up getting a little closer to the resiliency engineering side. And this is where my real incident response sort of career was born. There was no connection between what was going on with the platform, whether it be like a technical issue or an outage, and how we were communicating externally. So that was like the first task that I had was to build sort of a bridge between engineering and communications and customer support and social media and all of that to say, like, when a super technical incident is going on in incident room and everyone's looking and nobody knows what it means, here's how we communicate about that externally and internally.Here's how we tell support what's going on. So it was very focused on building processes, building communications that took that technical stuff and made it make sense outside of it, because Shopify is a very technical product with mostly a non-technical audience. And that is when I got really interested in the technical side and reliability and what was actually happening with a platform and how our infrastructure worked and what it meant to be in the cloud. And it was my first introduction to, oh, there is physical data centers and how does the Internet work? And from there, I just dove deeper and deeper and deeper into that and learned a lot about how giant, complex system works. And I'm by no means an expert in infrastructure, but I've gotten to learn a lot about it, and that's what sparked that interest. And I've just continued learning from there, I guess. Oh, and then I landed at Rootly, so that's where I am now. I won't skip that part.I'm now a Reliability Advocate at rootly, so I get to help other companies solve similar problems that I've solved in my career and talk about reliability and incident response and things that I've learned and meet other people who are interested in it and just kind of build a bit of a community around that space. That's so cool.ADRIANA: And it's interesting, too, because all of the previous experience that you'd had had brought you to where you are now.ASHLEY: Right.ADRIANA: I think even the stuff that you were doing at Disney kind of gave you that empathy for the customer that is so important when it comes to reliability that we don't talk about enough.ASHLEY: Yeah, I think even if I go before that, it's one of those things where, in hindsight, all of these things that I've done started to make sense. Like when I was a server at a restaurant, I was always the girl who would go to the angry tables and help move things over and comp the meal or talk to them, talk them out of leaving us a bad Yelp review. And I sort of found at some point that combined with software, and it was like this whole new world opened up where that was still a thing. But there was also a lot of new stuff for me to learn in terms of how complex systems work and the infrastructure that powers the Internet and apps. And I just found that so interesting that I couldn't stop learning about it.ADRIANA: Yeah, it's ridiculously complex, but it's also incredible to realize that even the job that you mentioned as being a server and having to deal with angry customers, I mean, there is no better test bed for being in reliability than to deal with angry restaurant customers because that can be really scary.ASHLEY: Yeah. And I realized that that had value that I didn't always see at first. I think I saw that as sort of a, like, that's my former life from before I had a real job in tech and everything. And then at some point I realized not everyone has that experience. Not everybody's had to do that all night and talk to people face to face who are mad at you. And that builds a lot of communication skills that not everybody has. So I found that that was something that I could bring into a world that I was very new in, like tech, and still have something unique that I was bringing to the table. And then eventually I kind of transitioned out of that more customer focused side to operational and more technical as I went along. But that continues to be very useful. I don't think there's a situation in life where the ability to talk to people stops being helpful.ADRIANA: Yeah, that's so true. Yeah. And I think also there's this misconception that in tech you don't need to have those soft skills, especially if you're like a software engineer, for example, that as long as you can code, then that's all that matters, or whatever it can be. Even for an ops person, as long as you make sure that the systems are up and running, it's all good. But it's not. You have to be able to communicate as a software engineer. You have to be able to communicate your ideas beyond just the code. As an operator, you have to be able to also communicate your ideas beyond just operating those environments.And I wish there was a little bit more emphasis put in communication in our education system because we all like, you know, I, when I went to school, we had to take a technical writing course and everybody just freaking groaned at having to take this technical writing course. And I hated it too. It was so dry. But it was useful, right, because you need an effect... Technical writing teaches you to communicate in a very effective and efficient manner, which is important in our industry.ASHLEY: Yeah. And then if you look at reliability and incident response specifically, it becomes even more important against the backdrop of those situations and the pressure that people are under. And the technical skills are also important because those are some of the most technically complex situations you run up against. That in combination with somebody might feel some type of way about what's broken. And it could be your customers, but it can also be your support team and your marketing team that had a big launch that day and the exec team that doesn't understand infrastructure, but wants to know in vivid detail exactly what's happening, but in like ten words or less. And you're like, "AH!"ADRIANA: And that is a skill. That is a skill. Speaking executive versus speaking to your peers or to your manager. Like, it's different language altogether.ASHLEY: Oh, yeah, I wrote a blog post about that recently. That's something I learned a lot about in my time, especially at Shopify, working with execs and managing incidents and realizing like, oh, yeah, you need to be very intentional. And it's not because execs are mean and scary. It's because they're looking at things from a totally different vantage point. They understand the business differently. They have so much context that you don't, and they have very little time. Their time is very accounted for, very expensive. You need to learn how to be effective.ADRIANA: Yeah, and the other thing that I learned was being able to speak in dollars and cents also goes a very long way with execs because they want to know, like, yeah, this is great, but what do I get out of this thing? So you want to buy this new whatever. So what?ASHLEY: What do you need from me? How much is it going to cost?ADRIANA: How is it going to help in dollars and cents?ASHLEY: And, yeah, exactly.ADRIANA: I feel you. Another topic I wanted to touch upon because you mentioned that you worked in the reliability space at Shopify. When you and I chatted earlier, you'd mentioned that you'd been on-call. Can you share some of your on-call experiences with folks?ASHLEY: Yeah, I have been very on-call for a long time. Like I said, it was just the three of us in kind of the earliest days of instant response, and we just threw ourselves on a pager thinking this will be like the easiest way for anybody to get a hold of us. And there were a lot of late night wake ups, and we had very little process built around what to do when the pager goes off. It was just kind of, if you get paged, you figure it out. And eventually that scaled. And we learned to manage expectations, manage what qualifies as a page versus like, send me a Slack ping and I'll deal with it in the morning. Maybe something that really comes to mind as a very intense version of the on-call experience was the Black Friday Cyber Monday preparation cycle and weekend at Shopify. Just because ecommerce is the highest pressure weekend of the year, and I think every Black Friday from 2016 or 17 to 2022, I spent 96 consecutive hours on that Black Friday pager ready at any moment.And I learned a lot, honestly, I think it's necessary to put yourself on a pager even if you're on operations at some point, because when you're building all of those processes and plans of what's going to happen, when you know that it's you that's going to get paged, you care so much more and you're a lot more thoughtful and you have the experience and context to say, like, yeah, this thing looks good on paper, but that is not going to work. I think that's a common mistake that I see in incident management and process, where people want to prepare for a specific scenario with a specific playbook that they will then execute. And sadly, it just almost never happens that way. So people overrotate to process a little bit and think, like, we want to get to a point where you can blindly follow this process when something happens, but that's just not the reality. So I think the most important thing, or maybe I don't know if it's the most important thing, but I think any company that has a pager should invest more time into talking about the on-call culture at their company. What it means to be on-call, what's expected of you and what's not expected of you. Should you be glued to your laptop for that entire time? Can you go walk your dog? In my opinion, you should be able to do that. You should have reasonable expectations.Like, get to your laptop within 15 minutes, respond to the page within five is a good benchmark example. But, yeah, I could talk about on-call for a long time because I've done so much of it. And at Rootley, that's like one of the biggest things that we're on a mission to do is just make that experience better for people.ADRIANA: Yeah, because, I mean, it can cause some serious PTSD.ASHLEY: It can. It can cause very real stress and in some cases, probably even actual trauma. I had a great experience. Shopify is an amazing place to work. It's a great culture. So I won't say that it was traumatizing, but it was stressful, for sure. And those of us who went through those sort of early days of it, once we had built up that empathy, we put a lot of effort into onboarding to prepare people for what it's like. But you also kind of want to balance that with making it better.So we would talk a lot about on-call health, and especially in the lead up to major events like Black Friday, we would have a lot of messaging around on-call and wellness. Like, are you checking in with yourself every hour to make sure you have eaten and you have taken a screen break and you've gone on a walk, or do you have a bottle of water around? Just those little things that can remind people. And that's something that I've actually encouraged some customers that use Rootly to build into some of their automation for incident commanders because we do have a little prompt that can pop up when you're assigned a role in an incident. So say you're assigned incident commander. It might say, here's your responsibility. But I usually encourage people also say, put something encouraging in there to say, take a screen break. If you have to lean on your secondary on-call, you're not in it alone. And that's another thing I feel strongly about, is it should be illegal to have an on-call rotation with no secondary.ADRIANA: Yeah, absolutely. Because that way you feel like you're supported. Like if shit hits the fan, you know that you can lean on someone else.ASHLEY: Yeah. And life happens. Ideally, for an on-call shift, you're available, you're on-call, but what happens if your dog breaks its leg and you got to go to the vet? Things happen and you need to have some sort of backup plan. Single points of failure are bad in software, and they're bad in people systems, too.ADRIANA: Absolutely. And you touched on something that I thought was so important, which is really building up that culture around incident management, because, as you pointed out, you can try as hard as you can to account for every single little thing that will happen with your system, but reality strikes, and you're mostly dealing with those unknown unknowns rather than the known unknowns. And so I think being able to mentally prepare for it, and I guess being also in a psychologically safe place where you can actually troubleshoot in peace is super.ASHLEY: Yes, totally. Yeah. I think there's, like, an element of protection that SREs need, and I feel pretty strongly about having not just SRE commanding the incident and also trying to fix it, but having an incident commander whose job it's not to fix what's broken, it's to protect the responders and keep things on track and keep those distractions, like that exec that storms into the channel, that's like, the incident commander should be like, whoa, let me stop you there. I'm going to field this. You're not getting anywhere near our on-call engineers. They're working on the problem. It's not their job to explain to you why this happened when we haven't even mitigated the problem yet, let alone found a root cause, which is like a whole other thing but people who don't understand the technical aspects, which is fine, you don't have to understand it, but there's kind of that healthy boundary and respect to say we're not there yet. We're currently investigating the problem, and here's where you can get an update every 15 minutes, and it's not in this SRE's DM, so back it up a little.ADRIANA: Yeah, as you said that I was getting flashbacks to earlier in my career of being on a call during a major issue where there's some friggin exec who's, like, poking their nose into your business and, oh, well, I used to code 20 years ago, why don't you restart the database? And it's like, buddy, back off.ASHLEY: Yeah, you really get it from all sides. And incident response, too. I mean, I've seen outages where you get the people coming out of the weeds on Twitter...X, or whatever. I was in it for eight years, and I could have resolved this in five minutes, and you're like, please.ADRIANA: Get off my back, buddy.ASHLEY: Yeah, totally. It's wild. It's a lot of pressure. So I think companies owe something to the people who are there solving some of the worst, most urgent problems to make sure that there's a culture and process and tooling in place, that it's not harder than it needs to be because it's already pretty hard, even with all that stuff.ADRIANA: Yeah, exactly. Like, you're stressed, you're in the middle of an incident, there be problems.ASHLEY: If you don't want to burn through your SREs, then you got to make the experience bearable.ADRIANA: Yeah, absolutely.ASHLEY: Sorry. No, you're good.ADRIANA: Yeah, I was going to say, I remember my previous job. I was managing a couple of teams, and one of the teams I managed was a platform team, and we had an on-call rotation because we're managing the Hashicorp infrastructure. And part of it included, like, I had a sub team of SREs and one of the guys on the team, he had been in operations for most of his career and had had some very traumatic on-call experiences. So even the thought of him being on-call, he was very much not down for it. And I couldn't blame him for it either, because that is some deep-seeded trauma that can be very hard to get rid of, to overcome. And I don't know, maybe being in a more welcoming environment can help them heal, but maybe there are some wounds that are just too deep where you just kind of have to avoid those types of roles, if you've been in such a traumatic spot.ASHLEY: Yeah, I'm sure that does happen. I will put a positive spin on it in that I have kind of a nice story, I just remembered, about an example of that culture really existing. And for a time when I was at Shopify, our CTO was Alan Leinwand. He's now the CTO at Webflow, and they just so happen to be a customer of ours. But that's not why I'm shouting him out. I am shouting him out because I vividly remember an incident at Shopify where the engineer who caused the incident, it was just...you know...he had shipped a PR that broke something. It happens. And he felt so bad and he was in the incident room channel, which...Shopify is a big company.There's thousands of people in this channel watching this. And you can tell he's flustered and he's embarrassed and he's saying, like, I'm so sorry. I should have tested against this condition and I didn't, and I'm so sorry. Know, you guys are all having to deal with this, and I'll stay late and blah, blah, blah. And Alan, who I know was always paying attention to what's going on in incident room and had a large amount of trust for the engineering team to handle things. He wouldn't jump in and start bombarding everybody, but he just dropped a message, a very discreet threaded response on that engineer's message that just said, like, "Hey, it's okay. You did the right thing. You noticed something was broken.You paged the on-call team and every great engineer has broken things. It's not what you break, it's how you fix it." And just gave him a really nice reassuring...and didn't make a whole "@here I'm the CTO. Look at me. Praising." It was just tucked away in a thread, just like some words of encouragement and reassuring him, this is totally normal and it happens and you're fine. And I think that was just like such a nice example of how the culture can be if you actually have people who understand instant response and just have empathy for people. Because engineering is really freaking hard. That was nice.ADRIANA: It's such a nice story. I love it so much.ASHLEY: So shout out Alan. Great CTO, in my experience.ADRIANA: We need more folks like him. Absolutely.ASHLEY: Exactly.ADRIANA: One other thing that I wanted to ask, because now you've gone from Shopify, big huge company, to Rootly, startup, how has it been? What do you notice in terms of going from a really large company to a really small company?ASHLEY: Yeah. Oh, my gosh. It's been crazy. It's a huge adjustment. When I had left Shopify, I didn't really know what I was going to do. I didn't have much of a plan. And JJ, our CEO and co-founder at Rootley, approached me and was telling me about the company and know role he wanted to build. That was kind of a developer relations-style role, but really focused on reliability and incident response. And I'd never been a DevRel. I didn't even frankly fully know what that was. But he wanted somebody who had been on-call and who had done the work. And a just hearing his story of how the product came to be, that he was solving similar problems at Instacart with our co-founder Quentin, who was their first SRE, I just felt like he really understood the reality of what it was like to work in incidents. And when I looked at the product, I really just loved the product. I, in my previous role at Shopify, had looked at incident management tooling. This was a while before Rootly had come on the market. And spoiler alert, we built our own because we just couldn't quite find anything that fit what we had wanted at the time.And when I saw Rootly, I was like, finally someone gets like, this is what I was looking for, not some big, clunky, over-engineered standalone platform that I'm never going to be able to get anybody to adopt. That's going to take months of development work to just even get up and running. This is so simple. It just plugs in. We can use it in Slack. So that was the first thing for me, was just really liking the product in terms of transitioning to a startup. Those first three or four weeks, I was like, oh my God, I don't know if I'm going to make it. The adjustment of the pace was crazy.I just couldn't believe how fast they were shipping. I thought I worked fast in my previous jobs. I thought we had a fast pace. That is nothing compared to a true Series A startup grind. But once I realized that it wasn't impossible, I just had to shake off some of the big corporate rust and stuff that I had in my system. I'm like, well, what do you mean? We're not running this through five different teams for approval. You trust me to just do it and ship it? And it was like, yeah, if you think it's going to be cool, just ship it. So it was just a massive increase in the amount of autonomy and the pace I was working at and also the amount of creativity.I think that's like one of my favorite things is we don't have the brand guidelines that you have to adhere to. And this is what we say and what we don't say, and this is how much we spend on this. When you're at a really large scale, there's a lot of process and rules, and they're established for good reason, because when you have 15,000 people, you got to have them marching in the same direction or else it's going to be a disaster. When you have 25 people, you can be really tightly aligned without all of that friction. So that was just like a breath of fresh air. And now I feel like I've really hit my stride with it and I'm less scared to move as fast as we move. It was a little scary at first. It was like I was standing on like a freeway and cars are just like whizzing past me and I was like...AHHHH!ADRIANA: And now you found your groove?ASHLEY: Yeah, now I'm good. I'm having a lot of fun. It's a great team. Everybody cares so much and is so fun and passionate about what we're doing. And a lot of people have experience doing this, too, like Ryan, who is on our post-sales team. He has done similar jobs to what I did at Shopify, but he was at Twilio and these other companies. So having people you can really geek out about, what was it like building incident response at a hyper growth company? And you're like, oh, my God, this was so hard. And this was so fun and I learned this. It's just a really energizing group to be a part of. So it's super fun.ADRIANA: That's awesome. Well, thanks for sharing. Well, we are coming up time. And before we go, do you have any parting words of wisdom on incident response or generally anything tech related that you would like to share with our audience?ASHLEY: My parting words are, if you are curious about incident response, becoming an incident commander but you are scared to get started, I really recommend you reach out to somebody who's doing it well at your company and ask if you can just shadow them and watch what they're doing and learn from them because it's not as scary as it looks, and you can totally do it if it's something you're interested in. And you don't even have to be all that technical because I sure wasn't. You can just learn. And of course, I have to give my little Rootly plug. If you want to learn how to make life better for your on-call responders and just streamline your entire incident response process, check out Rootly. We automate incident management in Slack across tons of different integrations. Whatever it is you use, we integrate with it and it just makes managing incidents so much easier. So check us out.We're at rootly.com. We'll give you a free demo. It's free to try. You could even set up a trial, no credit card. Start playing around with it. And we do tons of events. So if you're heading to KubeCon, maybe in March in Paris, I'll see you there at the Rootly booth. I don't know. I'll be there.ADRIANA: Well, thank you so much Ashley, for geeking out with me today. Y'all don't forget, subscribe and be sure to check out the show notes for additional resources and to connect with us and our guests on social media. Until next time...ASHLEY: Peace out and geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. It be sure to follow us on all the socials by going to bento.me/geekingout

About our guest:Nayana Shetty is a Principal Engineer at the LEGO Group. The LEGO Group is going through a massive digital transformation and she is helping with the architecture and engineering practices especially in the Ecommerce, Marketing and Channels technology. Over the years, she has led teams building products and tools that help organizations with site reliability and getting on the devops journey. Starting her career as a Quality Engineer, she is passionate about building quality into products from the start rather than an afterthought and creating a culture of quality using DevOps practices within teams.Find our guest on:X (Twitter)LinkedInFind us on:All of our social channels are on bento.me/geekingoutAll of Adriana's social channels are on bento.me/adrianamvillelaShow Links:The LEGO GroupTescoDomain-Driven DesignThe Financial Times (FT)Filling the Jar of Impact and Trust as a Principal Engineer (talk by Nayana Shetty)The Four DORA MetricsTranscript:ADRIANA: Hey, y'all, welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela. Coming to you from Toronto, Canada. And geeking out with me today is Nyana Shetty from the LEGO Group. Welcome, Nyana.NAYANA: Hi, Adriana. And I'm excited to be here and it's going to be interesting to see what we uncover over the next half an hour or so.ADRIANA: Yes, absolutely. I am super stoked because I always love talking to fellow women in tech. And also you work at Lego, which...so iconic! Such an iconic product!NAYANA: I mean, there's not been one person where I've introduced myself and said that I work for the LEGO Group and let's not put smile on them. There's not been one. I think it's the best place to work at that way.ADRIANA: I can totally imagine. Well, before we dig into that, I've got some lightning round questions to ask you. They will be quick and painless.NAYANA: Okay, let's try.ADRIANA: All right, are you ready? Let's go. Okay, first question. Are you a lefty or a righty?NAYANA: Righty.ADRIANA: All right, do you prefer iPhone or Android?NAYANA: iPhone any day. I don't know how Android works anymore.ADRIANA: Fair enough. Do you prefer Mac, Linux or Windows?NAYANA: Mac for most of my day to day work, but if it's actual tech work, Linux any day.ADRIANA: I hear that a lot actually. Very cool, very cool. What is your favorite programming language?NAYANA: Python.ADRIANA: Me too.NAYANA: I've done Python for a few years now and I don't write in Python anymore because I'm mostly in marketing and channels technology area where it's more about providing services for marketing use cases. And there's not a lot of Python there. But my love is always for Python.ADRIANA: I feel you. Yeah. I always tell people I'm happy when I code in Python. I'm like e best language ever.NAYANA: It's so much more easier than callbacks and Javascript. Oh my God, yes.ADRIANA: I'm sorry to people who like Javascript. I don't get it. I'm sorry. Okay, next question. Do you prefer Dev or Ops?NAYANA: Can I go DevOps?ADRIANA: Yeah, that is a perfectly valid answer. A lot of people have answered that. Totally. Next question. JSON or YAML?NAYANA: JSON.ADRIANA: Oh, interesting! I always find it funny when people who are Python lovers say JSON because you're already forced to indent. You get bitten by the indentation bug in Python anyway, so that's very interesting. All right, I just added this one today because it came up in another recording that I did. Spaces or tabs?NAYANA: Spaces. I think less confusing than tabs. When you have spaces and tabs, it's confusing. So just keep it simple. Just do one thing. Spaces.ADRIANA: Yeah, I totally feel you. I'm curious after I start asking this question more regularly, like what other people are going to respond. So thank you for being my first respondent to the spaces versus tabs question. Two more questions. Do you prefer to consume content through video or text?NAYANA: Video.ADRIANA: Interesting.NAYANA: I learn a lot by seeing and listening rather than just reading. So yeah, I think my preferred option is videos and second option is listening and then the third option is reading.ADRIANA: Right. That's so interesting because I think you're like the first person in a while who said video. So one pick for the video camp. Awesome. And final question, what is your superpower?NAYANA: What is my superpower? I think being extremely structured, that's my superpower. I can convert any problem into logical steps and say, this is what that.ADRIANA: Is a good superpower to have. I do find like whenever I'm in a position where all the thoughts are in disarray, like just sitting down and doing a list is like, magical.NAYANA: My role kind of calls for it a bit because I kind of tell people a principal engineer's role is to go into places where you don't know anything about the topic. You go discover, find out, and then you know, but also you get the rest of the organization know what and how to move forward with and when you're going with so much uncertainty, if you had some structure, I think it's much more easier to work through it.ADRIANA: Yeah, that makes so much sense. That makes so much sense. And I think that is probably like having that skill set is probably the most valuable skill set for a software engineer. Right. Because we're constantly encountering these scenarios that we haven't encountered before.NAYANA: Yeah. And I've seen so many times when engineers get, as soon as they see a problem, they start digging into it and they're like, oh, I'm going to solve this with arrays or lists. And I'm like, what are you trying to do? Step back and just come up with a plan. And it doesn't have to be like something that you set in stone. Keep it fluid, but at the same time have a plan.ADRIANA: Yeah, it's true. It's funny, I remember when I was a kid, so my dad is like, he's a math guy, he's a software guy, and I used to come to him to ask for help with math problems. And he would always bug me. He's like, do you have a plan? That's been permanently etched in my mind now. It's like, you need a plan of attack for solving this problem. Where is it? You can't solve this problem without it.NAYANA: Yeah, I have a story from, really from two weeks ago now where we do all of these planning, and then you put four people in a room and say, you've got ten minutes and you have to solve this puzzle. All that goes out of the window, though, you start thinking about, oh, how do I solve this problem? In the process? When this happened two weeks ago for us, we forgot that we had to collaborate with two other teams that were sitting in two other rooms. We were like, if we had just stepped back and come up with a plan, we would have won this challenge.ADRIANA: Yeah, so true. So for all you out there, planning goes a long way. So I wanted to take a step back because as we mentioned in the outset, you work for LEGO Group. I mean, I'm super curious, what does it mean to work in technology at LEGO Group? Because we always think of, like Lego as the toy, the physical bricks that we put together. So what does that look like?NAYANA: So there's a lot of digital transformation that's been happening over the last two years or two to three years at the LEGO Group. And this means that there's technology being introduced in all sorts of places from, like, I work in the marketing and channels technology area, which is to do with how do we sell to shoppers on Lego.com to how do we sell on Amazon and Tesco and all of those kind of places, to how do we do good marketing? And Lego has a very strong brand image. How do we sustain it and how do we build on top of it? What sort of technologies can support those kind of brand image and the shopping side of things. So that's with the marketing side of things, which is where I mostly work on. I do very little on the product side of things. But then there are parts of the organization that look at the kids experiences. We've split ourselves into shopper, partner and consumer. Shopper is someone who buys from us, mostly adults.And then there's partners who are like Amazon and Tesco, those kind of companies who buy from us. And then there's consumer, which is actually kids and adults who actually play with our products. And the experience they have is very different to what, when you're buying a product should have. And how do you bring technology closer to them, especially with so much digitization happening and Lego bricks it's still something very physical. How do you bring technology into something so physical, I think is an interesting challenge. And then there's the whole operations and the supply chain and side of things where there's the manufacturing of start with the planning of creating the products to how you then manufacture it, and then how do you ship it and all of those things. There's a lot of investment that's gone over the last few years in digitizing a lot of these things and bringing the business processes closer and redefining some of our business processes to be more engineering focused or simplifying it so that the architecture is much more simpler. And that kind of stuff has been a massive thing over the last few years now.And a lot of teams are...it's a new space for a lot of teams. When I joined two years ago, I was fascinated and surprised by how much you can push tools like SharePoint...and Microsoft Sharepoint and Excel and PowerPoint to run a business. Basically, I came from an organization where everything was digital. So for me this was fascinating to see that they're actually selling, planning, selling and all of those just through Excel sheets. And now, two years down the line, we see a lot of digital services which are actually solving these for our business. And yeah, I think that's where engineering and technology plays a very strong hand in how we move forward as the LEGO Group and how we evolve ourselves, I guess.ADRIANA: Right. And so what do you think right now are some of the most challenging problems that you're working on?NAYANA: So the area I look at in marketing is to do with personalization. And because of the strong brand we have had as the LEGO Group, we didn't have to go to the level of individual persons needs and requests to actually figure out how do we make a difference in their shopping experiences. Until recently, and now we've pivoted to be like, it's all about that experience, especially Gen Z. And the future generations are so much on the Internet that everything they need has to be personalized. And there's an expectation that if you don't know me, don't sell things to me. That's how I think the expectation is. So one of the major challenge I'm working with is how do we bring technology into personalization? How do we collect data in a much secure way. So there's the whole legal and privacy aspects of collecting personal data.And then how do we then translate that into making sure that we use it in a consistent way across our different product teams and stuff. So one of the things that we hold quite dearly from a principles perspective, is that we follow domain-driven design thinking, which means that there's very modular, clear boundaries to our product teams and they can work independently to deliver the business outcomes in the marketing space. That's actually a not so common concept. When you look at any tools that are out in the market, they're very much like they can solve it all for you in one single product, but you don't need one product for the whole thing. You have four different product teams looking at it. So how do we break that? Like a single monolith kind of approach, which is what marketing has been in the past, to much more modular, domain-driven kind of product themes and product areas and stuff. So that's been one of the major areas that I've been working on over the last year. The other one, which has started cropping up more recently, is how do we collect or gather engineering metrics? And this comes from the fact that we've invested a lot over the last two, three years in technology.We've grown quite a lot. How do we know it's actually bringing us the right return on investment? And what are the right indicators that show us that our engineering teams are working efficiently and stuff? And we need to do this in a way that's not poking individual teams saying, oh, you're better than them, because your, let's say deployment frequency is five and that team's deployment, that's not the level we should be going into, but what is it that we should be looking at more widely? So that's another area that we've been exploring quite heavily more recently. And I think this will be a hot topic for the next year as well.ADRIANA: Right, right. Now, pulling back a little bit, because you mentioned you're a principal engineer, you touched upon some of the things that are within the purview of your responsibility or the expectations as a principal engineer, what would you say is kind of the one thing that stuck out for you more than anything when you moved into a principal engineer role? Because the expectations are vastly different from, say, a junior, where you're like, you're just writing some code that someone told you to write.NAYANA: Yeah. So I moved into a principal engineer role. I was at the FT when...Financial Times...when I moved into the principal engineer role, and for me at that point, it was like, and principal engineer roles are very different across different organizations where at the FT, it was a 50/50 kind of role, where 50% of the time I spent on people management and team health and that kind of stuff. And 50% of the time is what I thought about tech strategy and the direction we want to move in and stuff. Where now at the LEGO Group, it's very much the tech strategy role. It's an individual contributor role where I'm mostly thinking about the long-term direction and the guardrails that we need to enable the teams on. And I think what surprises me and what had surprised me, and probably it's something that anyone who comes in new to this role will have to work with, is how much hands-on experience do you want to have in this role? And I think it varies. And in the LEGO Group, we've got around 15-ish principal engineers, and each one of us have a different version of principal engineering that we do. So it's not the same role that...as a role or as a level...it's the same level that all of us have.The roles that we cater to within the organization is subtly different based on what our strengths are. I think where some people are very much into, oh, I'm an expert in, let's say, for example, a technology like SAP. I'm an expert in SAP. So I'm going to be spreading across wherever SAP expertise are needed. And I'm going to do some hands-on supporting or even the strategic thinking around SAP. Where I'm more of a solution architect kind of principal engineer, where I do very little hands on. I do hands on just so that I remember and stay true to the title of engineer. But my role itself doesn't want me or it doesn't need me to actually do a lot of hands on coding and that kind of stuff. So I think that's something that surprised me thinking, oh, I thought principal engineer is going to be like the smartest engineer in the room, which is not true.I'm not that I kind of see myself as a person who can glue the right people together so we reach that best outcome possible for the organization.ADRIANA: Right. And that is such an important skill. I mean, it's not necessarily about having the answers, it's knowing the people who have the answers and putting them together.NAYANA: Yeah. So I kind of see myself doing the glue. I say principal engineers are the glue across the organization. Breaking some of those silos and barriers across organizational constraints and stuff. That's what a principal engineer should be looking at. And then the other thing is we're also part of leadership team. So I'm part of the marketing and channels technology leadership team. So what sort of engineering culture do I want to help the organization get behind? And that kind of stuff and being, like, a positive influence on it.I mean, given my role, I work very closely with engineers on a day-to-day basis. So I hear a lot, like, just on the ground kind of, I wish they had this. I wish we had done that. So just hearing those things, and when there's enough of those wishes that you hear, you're like, okay, can we positively influence it? And that's, I think, something that a principal engineer kind of plays a key role in bringing that people's voice into spaces where there's just leadership...leaders in a room and stuff. And how do you then create and change because of what you know and what you're surrounded by and stuff?ADRIANA: Yeah, that makes a lot of sense. And it's interesting because I think having your ear to the ground, knowing what's around you, I think it helps. As you said, you're bringing the challenges and the needs and wants of the other engineers to the forefront. But also, I guess it helps you with that glue aspect of your job as well, because then you're in tune with, like, who knows what.NAYANA: Yeah, exactly. And also it adds to they trust me enough because I help them somehow kind of thing. How do you build trust, especially? And this was something which I think was something that I had to learn when I moved to the LEGO Group, because at the Financial Times, I moved through different roles. And I started as a QA Lead and then moved on to Tech Leading and then slowly into being a Principal Engineer, where people have seen how I contribute and what my opinions are about stuff, so they already know about me, and they trust me enough based on what I've delivered. Where when I started here, it was completely different. Where some people trust. I mean, there's an inherent trust that you get because you're in a role, but other than that, there's not trust that they believe you. You've done that thing. That's why I trust you.So how do you generate that quick trust among your peers and people who you work with is something that I had to learn as part of joining the LEGO Group. And, I mean, there's probably something that the LEGO Group is really good at, which is that open culture of, you can just go and talk to people, understand where they're coming from, and then it's one of those places where I felt it's easy to debate, but then at the end conclude somewhere.ADRIANA: Cool, because I think you touched on something really interesting, because I think one of the hardest parts about joining a new organization is having to build up that reputation, that trust, so that people see that you're worth the paycheck you're earning. And that can be really scary, right? Because you've got that ramp up time where you got to figure out the landscape, but then at the same time you have to have some level of productivity so that people are like, okay, I know I can go to her. I trust her.NAYANA: Yeah, I spoke about this in a conference recently as well at the LeadDev in London, where I talked about these different sizes of problems as a principal engineer that you should be thinking of. So there's the whole, an analogy that I learned from my previous manager was the rock, pebbles and the sand, where if you fill your jar with sand first, then you have no space for the pebbles and the rock. So when you start in a new organization, the first thing worth doing is understanding what those sand are, what those rocks are, what the pebbles are that you can be getting yourself involved in, but being very conscious about what you can pick up. Because if you end up picking all of the sand, then you're doing all of those little changes, but nothing, it doesn't justify the salary you get. So how do you then rebalance to make sure there's enough rocks that you work on and pebbles and stuff. Something that you consciously think about.ADRIANA: That's a really great analogy. It's the first time I've heard that, but that's really good. The other thing that you touched upon that I want to dig a little bit deeper into, which I thought was something that I think in a lot of tech circles, there's this expectation that if you move into a management role that is equivalent to a leadership role, and that management is like a natural promotion cycle for whatever. I definitely have that impression. Like, when I started my career, I'm like, oh, yeah, I need to be a manager. I can't be a developer my whole life. No. But the thing that I thought that was really cool about what you said is that even in your position as a principal engineer, as an individual contributor, you're not managing people. But you have a very prominent leadership position.And I think that's such an important thing to underscore because I think a lot of people conflate like, oh, the only way to be a leader is by having a management position.NAYANA: Yeah. Doing this role now for two years, I think I found that this is indirect leadership, where you don't actually manage people, but you still have an influence on what happens within an organization. And when you're having that indirect leadership, it's all about how you bring along people on the journey and how do they, for me, when I'm working on a tech strategy or any of those things, it's the day when I hear other people tell that this is the tech strategy that we've got within our org. That is the day I feel like I've actually done my job because that's my role where I've influenced people enough that they have bought into it, that they call it out as the thing that needs to happen and stuff. So it's very different to like, if I was managing people, I could say, you report to me, and if you didn't do this, then we will have to go through all of the people management side of things, which I have none of those to do. So you do get the best parts of management, I think, being an individual contributor and influencing the rest of the where you're bringing people along. But at the same time, I do work very closely with the senior directors who actually have the management responsibility of these orgs where if I see things moving slowly, I kind of lean on them saying, can you help nudge this happen or can you help make this happen within the organization? And they surely have the direct influence, which I don't, but you have to use that levers at times, but at least 80% of the time you can get away without that.ADRIANA: Yeah. And I think that's an extremely important and useful skill to have no matter what right to be able to exert influence. So for you, what's your strategy in terms of exerting influence? How do you make it work? Because I mean, it's so difficult, right? Especially when you're dealing with all kinds of people, people who are like, oh, this is the best idea ever. And then there's the, no, I don't care, I don't like your idea. Not gonna do it.NAYANA: It happens. And I think this is where it's the carrot and the stick kind of approach where you show the carrots. And then there are times when a lot of the work that I do is talking about why it's important to do certain things in a certain way. Or if you're saying...recently I was working on what are guardrails around PII data handling is and getting the right people involved. So it was not just engineers opinions, but getting people from legal and privacy office involved in that decision making from the start. So we're not bringing them later on. But when we thought about, this is an idea that we should do something about, bring people on early so they feel like they have contributed into it and they have a stake in it, which sometimes can be hard given everyone's got busy lives and there's a lot of people working on product teams are thinking about, oh, this is my OKR that I have to deliver to and all of those kind of things.So taking away from that is what a lot of principal engineers will have to do where we say, oh, that is important. But in the next quarter, if we did this, this will make your life easy, like just showing that futuristic view of what will benefit them. And also in a lot of times it's about coaching and mentoring people through. If you contributed through this process, then you can get into...as you develop...engineering management is not just the option. You can also think about IC roles like we have as principal engineers and stuff. And when it all fails, that's when it's probably just getting back to the people leaders...of these people who...troublemakers. Are they troublemakers? I don't know. And then getting them to help a bit more.And this is where I kind of see that relationship between the senior directors and principal engineers being really close, where we work quite like if we have to influence without authority, we need their support when that authority is needed.ADRIANA: Yeah, absolutely. Sometimes it's one of those things where you can't and probably shouldn't handle all the issues yourself anyway. So it's good to know who you can call on to ask for help, to ask for the little nudge to get people around to your corner. I think that's also like an aspect of influence, right? Is having a circle of people who trust that you know what you're doing and will follow you because they like your leadership and that they can also exert their influence to influence others because they believe in what you do, which is super cool.NAYANA: I've also used other principal engineers across the organization as a support network because the kinds of problems I work with are not within my own product area scope. It can also span across multiple product areas, or we call it clusters within the LEGO Group. So when there are things spanning across clusters, I kind of lean on the other principal engineers and we work together so that it's more of like, it's not her opinion, it's the company's opinion kind of comes in, which also is quite handy to have in places when you have to influence people where they don't come within your part of the organization. They have no clue where you sit in the organization. They've not worked with you, so they don't trust you enough. So in those kind of it's worth calling in on your support network outside of your immediate organization and stuff.ADRIANA: Yeah, and it makes sense because, again, it helps to build that trust because it doesn't seem then like, oh, you have an agenda. No, there are other people who see this in the same way. So, hey, maybe there's something to it.NAYANA: And like, I think a good thing that the LEGO Group...that I've seen at the LEGO Group is that we have some core principles that have been outlined at the start of this whole digital transformation. Like I mentioned about the domain-driven principles that we follow. Or it could be that API-first kind of approach or like the Cloud-first kind of. I think the core principles that we've laid out, forms like that foundation that we can lean on quite heavily when it comes to, "Okay, I'm lost here. What do I fall back on?" I can fall back on those core principles, and they're not special for the LEGO Group. So you can read about it, how other companies are doing to actually learn from it. And then it's a good safety net to have, especially when you're starting new in an organization, having that kind of a foundation layer that you can fall back on and stuff is quite handy as well.And it's also like when they're stuck in those scenarios where you have an argument, you can then go back to the first principles and talk about, okay, what does theory say? And then work forward from there kind of thing.ADRIANA: Yeah. Makes a lot of sense. Now, going back to something that you mentioned earlier, which is you mentioned that you don't get to do a whole lot of coding as part of your role, but you still like to stay sharp. So what do you do to stay sharp with your coding skills? Kind of your go to thing.NAYANA: Yeah. I think more lately what I've done is within the principal engineers group, we have a working group where we do hands-on work. We've set up a couple of hours every week where we just do some hands-on work. So the most recent one was we were all learning gen AI, and learning didn't mean just go read stuff, but actually build something. And we did more like a hackathon kind of style thing. But we're not spending a day. It's just a couple of hours each week. And I think that's how I have...for me, that's the easiest way to keep on top and also feel like I'm staying up-to-date and also avoiding that impostor syndrome to kick in as well. You're like, oh, am I current? Am I what...trustworthy enough? So I kind of use those hands-on sessions that we've got internally where spend a couple of hours each week just coding anything. And currently, given the gen AI, seems to be like the current hot topic. So that's one area and the other area is around...I haven't worked much in the data space, and a lot of teams that I'm currently working with work with data a lot. So this is where some of this Python skills does come in quite handy, which is just trying how to crunch data with Python and that kind of stuff. So it's always related to what I'm working on, but with a slight slant on nothing related to what the LEGO Group needs. Just what I need to keep myself updated, kind of.ADRIANA: Yeah, that is such a great way to stay current. It's funny because I was telling someone the other day that my last role, I was a manager, but I don't feel whole unless I'm coding. So even in that role, I would carve out some time in my week to make sure that I was learning new things, because otherwise I legitimately got depressed if I wasn't creating something. And I think if you're a software engineer, it's just kind of part of your blood. So to be able to find any excuse to learn something cool and to get that hands-on experience and to learn something that's actually interesting so that it'll stick in your mind more, right?NAYANA: Yeah. And I think another thing that I remembered was, when I attend conferences, if there's an interesting piece of technology I would have seen, that's another place where I just go and play around with that for a few days. And conferences are my trigger to play around with a few technologies as well. So I kind of make sure I attend a few just so that that becomes the reason why I'm trying out stuff. At the end of the day, it's how we manage our time. And we are at a stage, we are our own time...leaders of our own time, right? So we have to see how we manage it. And carving out time is so important, especially as you grow in your career, you can go into that thing of, I'm busy, so I can't learn. It's a very easy, vicious cycle to go into, but staying aware of it, I think, is quite good.ADRIANA: Yeah. Like you said, carving out the time is super important. And not using, as you just said, being busy as an excuse. Because I remember an instance when I was doing the management role where I was getting really frustrated because we had these no meeting Wednesdays that was like my day to play around with stuff, right? But then I kept booking meetings on my no meeting Wednesdays, so I had nobody to blame for myself. So until I took control of my calendar and started saying no, because people would book meetings on Wednesdays, and I kept saying yes. So I'm like, "NO!"NAYANA: It's very easy to do it. And I think I had a colleague at the Financial Times who she had, like, a tally chart that she used to maintain for the month to just see how many days of just coding she's done. And it was so interesting to see how you can put some data behind this and it's not too hard to do it. If I wrote some code or if I read some code today, then I just put a...like, it's a tiny chart. You just put a line on a book, right? And she kind of said that that was really motivating her to keep true to coding always. Or, like, doing something hands-on doesn't have to be. Exactly.ADRIANA: Ultimately. I mean, what we do is a very creative line of work. And I think as creators, we like to create and it makes us whole.NAYANA: Yeah, that's so true.ADRIANA: Awesome. Well, we are coming up on time, but before we part ways, are there any parting words of wisdom that you would like to share with our audience today?NAYANA: I'm trying to prioritize in my head which one's the better one. I think I'll probably go to one which we are in a macroeconomic situation where everything's getting tight and there are companies that are struggling. And for me, this is a time when engineering efficiency and thinking about how we build more sustainable products becomes quite important. So I think...thinking about engineering efficiencies and trying to not in a sense of I'm going to measure the four DORA metrics or...not that way, but more of what can make my software more sustainable, what can I do to make it more maintainable so that I don't have to put energy once I've built those products and stuff is probably the key thing that it's top of my mind at the moment. And I think it will be more. It's going to take a larger space next year when we don't know where the world is going and stuff.ADRIANA: Yeah. So very true. I think those are really great words of wisdom. Well, thank you so much, Nayana, for geeking out with me today, y'all don't forget to subscribe and be sure to check the show notes for additional resources and to connect with us and our guests on social media until next time.NAYANA: Peace out and geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:Riaan has worked for Multi-National companies in Portugal, Germany, China, United States, South Africa and Australia.Certified Hashicorp Terraform InstructorHashiCorp Ambassador 2021, 2022, 2023Creator of Hashiqube - The best DevOps Lab running all the Hashicorp productsHashiCorp Vault and Terraform CertifiedCertified Hashicorp Vault Implementation Partner10+ years relevant DevOps experience with a strong focus on Automation and Infrastructure / Configuration in Code.Find our guest on:X (Twitter)LinkedInYouTubeGitHubBlogFind us on:All of our social channels are on bento.me/geekingoutAll of Adriana's social channels are on bento.me/adrianamvillelaShow Links:VersentTelstraUbuntu LinuxInstalling Ubuntu on Macbook ProMark ShuttleworthVSCode Dev ContainersHashiCorp Configuration Language (HCL)AWS CloudformationPuppetMagento%20and%20Symfony.)systemdHashiQube12 Rule for Life, by Jordan PetersonNever Finished: Unshackle Your Mind and Win the War Within, by David GogginsNSW Maritime and Road ServicesHashiCorp AmbassadorWiproHashiTalks 2024VagrantTerraformVaultRedHat Ansible TowerApache Airflow with DBTServianVault AssociateTerraform AWS EKS BlueprintsHashiCorp Core ContributorMitchell Hashimoto (HashiCorp co-founder)Armon Dadgar (HashiCorp co-founder)HashiCorp BUSLHashiTalks Deploy 2023TerragruntOpenTofuAzure BicepHira(HashiCorp) Boundary(HashiCorp) Waypoint(Windows) NT 4Gentoo LinuxVagrant Docker ProviderAnsible AWXTranscript:ADRIANA: Hey, y'all, welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability and everything in between. I'm your host, Adriana Villela. Coming to you from Toronto, Canada. And geeking out with me today is Riaan Nolan.RIAAN: Good morning, Adriana. How are you? It's good to see you. Happy Australia day. It's Australia day in Australia, so happy Australia day. At the moment, I'm working for a consultancy in Australia called Versent, and they've recently been bought by Australia's biggest telco, Telstra. So I'm a consultant for them. I do DevOps and HashiCorp stuff.ADRIANA: Amazing. So you said you're calling from Australia? Where in Australia are you calling from?RIAAN: I'm on the east coast in Brisbane. Brisbane, Australia, in Queensland. The state is called Queensland.ADRIANA: Awesome. And significantly hotter than the crappy rainy weather of Toronto today. We are at a balmy 3C. And you are at what temperature right now?RIAAN: Oh, my goodness. I'll tell you right now, weather. It's 25 degrees C right now...26 degrees C. It's 7:00 in the morning and it is going to go up to 30 degrees C today.ADRIANA: Oh, wow. Hey, my kind of weather, it's lovely.RIAAN: I tell you, it is so beautiful. We've got so many birds here, and thankfully I've got a pool here where I rent this property.ADRIANA: Oh, that's nice.RIAAN: If it gets too hot, I just jump in the pool.ADRIANA: That is very nice. Super jelly. Super jelly. That's cool. Well, are you ready for our lightning round questions?RIAAN: Yeah, sure. Let's see what you got.ADRIANA: All right. Yes. This is a get to know you better icebreaker sort of thing. Okay, first question. Are you a lefty or a righty?RIAAN: I'm right handed.ADRIANA: Awesome. Do you prefer iPhone or Android?RIAAN: I am on Android. I prefer Android.ADRIANA: All right. And do you prefer Mac, Linux or windows?RIAAN: Strangely, I'm the type of guy that used to run Linux on a Mac on my MacBook air. Yeah, Ubuntu.ADRIANA: Nice.RIAAN: Made by Mark Shuttleworth, who's from South Africa. But it just became a little bit difficult with all the changes. Work takes over. And so I've recently, well, not recently, about five years ago, switched to MacOS on a Mac.ADRIANA: Oh, nice. So you were running like Ubuntu natively on a Mac. It wasn't a VM, it was like...actually...RIAAN: I can't sometimes with the new stuff that doesn't work. But my old little MacBook Air that I got from Germany runs Ubuntu dual boot.ADRIANA: Oh my God, how cool is that. That's amazing.RIAAN: Because KDE is just such a great desktop. And it's got so many customizations and Windows gestures that it just makes your day to day and your working incredibly easy.ADRIANA: Very cool. And now you're like, no, now it's MacOS on the Mac.RIAAN: Now I've become not lazy, but when something breaks on my Mac because I work as a consultant, so I get a company PC and then sometimes I'm on Windows, sometimes I'm on Linux, sometimes on a cloud thing. So now I'm just the default OS with dev containers. So I use VSCode's dev containers, which means I just need VSCode and Docker and the rest I do inside of the container.ADRIANA: Nice.RIAAN: I really keep it so simple and so easy nowadays.ADRIANA: That's awesome. Hey, that is the way to do it. To keep it simple. We overcomplicate our lives. So, awesome.RIAAN: Yes.ADRIANA: Okay, next question. What's your favorite programming language?RIAAN: Listen man, I must come from systems administration. So I like Python and I like Bash and scripting. And then of course HCL is my favorite. And I used to start off with PHP back in the day on PHP, but I've since moved away from it. I used to do a little bit of PHP in Magento, but I'm just really in love with the infrastructure stuff and the DevOps. So I don't even know if you can call YAML and Cloudformation and HCL programming languages. You probably can't. So I'm a script kitty. Let's call me a script kitty, you know.ADRIANA: All right, I love it. Okay, next question. Related. Do you prefer dev or ops?RIAAN: I love both. And I really like the synergy. I used to do Puppet stuff, and when I discovered Puppet, I was like, wow, this is incredible. And then along came Cloudformation and I could just code something in Cloudformation and in the user data, pass it off to Puppet, and then do all of my stuff in Puppet. And that was the "Aha!" moment. We have finally arrived.ADRIANA: Nice.RIAAN: I like. What's that cake? A red velvet cake. It's a mix between the two and white chocolate, vanilla and chocolate. I love it so much.ADRIANA: Awesome! I love it! Okay, another one. And I think I have an inkling of what your preference is. Do you prefer JSON or YAML?RIAAN: To tell you the truth, I hated JSON when I started with Cloudformation, but it didn't support YAML. So I wrote so much cloudformation that I loved JSON. I started loving it. But what's more readable and easier for the users. I mean, I do like YAML. It is just so beautiful and simplistic and easy to read. So it's like your kids. Let's say I've got two kids. I love them both equally. The JSON is the kid with red hair and YAML is a beautiful dark brunette kid with hazel eyes. I love them both equally.ADRIANA: I love that. I love that. Now, what if you threw HCL into the mix...as a Hashi guy?RIAAN: I love HCL. It's the fastest growing programming language and you can use it everywhere and it's just so flexible and just so forgiving. The shorthand if else. It's just such a great. That's probably what I'm going to start my son off. He's almost ready to start learning something and I think I'll start him off with that because it's really powerful if you can write a little bit of HCL and deploy it, and there you've got ten virtual machines. Yeah, that will just be the thing I'm going to start him off with.ADRIANA: That's very cool. Speaking of programming languages, so my daughter is like a perpetual artist. Like, she's just born artsy and my husband and I are both in tech. And she was like, "I'm not learning how to code." And I'm like, "But you're a great problem solver. You would be a great coder." But I'm like, "I won't push it on you because you do you." And then she took like, I don't know why, but she took a computer class in school this year and learned Python.And she's like, and she's like, "Mom, I hate to admit it, but I love coding." And she's just wrapping up her semester and she's like, "I'm going to be so sad that there's no coding next semester because I really enjoy the daily coding challenges." And I'm like, that's vindicating.RIAAN: People always say, oh, well, you get the creativity kind and then you get the. But I really think that programming and DevOps stuff is a very creative art so much. It's not the boring essay type of stuff. And even the typing is also a creativity outlet. I really think there is a place for it.ADRIANA: Oh, yeah. And honestly, I think software engineering is such a creative profession. It's just creative in a very different way than. You're not painting on a canvas, a traditional canvas, but the IDE is your canvas.RIAAN: Yes. And you have to use your imagination when you run into a bug, you have to kind of walk it through and I wonder, what is it now? Yesterday I got a bug where HashiQube wouldn't start and I was like, is it the new Vagrant version? And then I'm like, what could it be? Could it be Docker? It turns out it's the Docker. The new Docker at 25.0 doesn't let Vagrant start. And you have to be creative. Where should I start looking now?ADRIANA: Oh my God. As a sidebar, let me tell you, every time there's a Docker update, I am like shaking in my booties because I feel like every Docker update causes my system to melt down and I can't run an update. I have to actually nuke Docker and then reinstall it and pray that other stuff that was relying on Docker is still working.RIAAN: And then yesterday with that bug, I go read the Docker change log and they had some problems with the systemd update. So the Docker developers must be like, every time there's a systemd update and I can't even just update it, I have to nuke my whole thing. It's amazing how dependent we are on each other's work. It's like this ecosystem.ADRIANA: Oh my God. Yes.RIAAN: It relies on other components.ADRIANA: Yeah, absolutely. Okay, next question from our series. Do you prefer spaces or tabs?RIAAN: I like spaces. I love spaces. Tabs give me that feeling where somebody walked over your grave. When I see it, I'm just like..."Ugh!"ADRIANA: That's awesome. That's such a great description. Okay, second last question. Do you prefer to consume content through video or text?RIAAN: That is funny. I'll tell you, I like video. I'm in two minds of what do we learn easier? I think text is too slow to make us humans learn. I love reading my book. I'm reading at the moment is Jordan Peterson's "Twelve Rules for Life". But I've been trying this out now. So while I'm reading it, I'm listening to the audiobook on Spotify and I don't know yet whether this is going to make it stick, but now I'm using my ears, my mind, and my reading, and I'm just now busy checking it out. What is going to be the best way to get content through your thick skull?ADRIANA: That is very cool.RIAAN: Learn it quicker. So I don't know, but I do like videos. I do love it when they give a link in the video to a GitHub repository. Yes, because it's like copying code from a picture. Copying code from a picture. I'm like,ADRIANA: Yeah, I know, right? Yeah. It's like, oh, I have to type this out.RIAAN: Anyway, that's where I am at the moment. Let's go with video with a link to a GitHub repo.ADRIANA: Awesome. I love it. You mentioned something interesting, which is like you're reading the book but also listening to the audiobook on Spotify. And I've done something similar. So I don't have too many physical books just because they take up too much space. But what I've done is I would buy the Kindle book but also get the Audible add on. So then if I was out for a run, I could listen to the book, and then if I was at home and in the mood to read, then I could open up the Kindle book and it would be in the exact spot where I left off in the Audible. And I'm like, oh, my God, this was like the best way to consume content, right? So for me, I thought it was so cool.RIAAN: Yeah. Follows actually your audio.ADRIANA: Yeah. Because they're tied the same. It's the same account, like the Audible account, uses my Kindle credentials. My Amazon account.RIAAN: Incredible. Yeah. I still have to have a little bookmark in the book.ADRIANA: Right.RIAAN: To keep it kind of in sync. Incredible. Wow. That's a good tip. I love physical books, but I might just switch now. I don't know. I'll let you know.ADRIANA: Yeah. My sister has a bunch of physical books, so she'll lend me one every so often. And I love the touch of a physical book. And there's something so satisfying about carrying a book around the house. But the convenience of the ebook is like, I can be like waiting at a doctor's office, open the iPhone and read my book.RIAAN: Yes, I do like it. I do like the physical mean. I've got a couple of them. Another great one is this one from David Goggins, and I was fortunate enough to meet him in person in Brisbane. And the other one I read before that was this thing. So weird, man. I mean, you know, after COVID, just as I was reading it, I was just keep on thinking how lucky and how thankful we are to be out of this COVID thing because they were going to pass rules from the World Health Organization and mandate us locally to countries and not all countries are the same. And I don't know, it was creating a sticky situation.So after this, I was just reading that book and every second page I was like, oh, thank God. I don't think I could have handled that one. So, yeah, I do like the physical books and stuff, but the Kindle is just so convenient.ADRIANA: Yeah, absolutely. All right, final question. What is your superpower?RIAAN: My superpower is probably, I'm curious and I'm quite patient. I can stick with a problem for a very long time. I might let it go for a little bit, but I would always come back to it and revisit it. And persistence is absolutely key. So I think that would be my superpower. I always say I'm not actually clever. My problem is that I'm curious. So through my curiosity, I just discover and I happen to learn stuff.RIAAN: I suppose. That's my superpower.ADRIANA: I love it. That's so great. Well, you've survived the lightning round questions. Awesome. Well, there's so many things I want to talk to you about, but one of the things, because you and I met when I was starting on my Hashi journey, where a coworker of mine found HashiQube, which you've created. And it is like whenever I have a chance, I will promote HashiQube to people, to Hashi folks, because I think it's such a great tool. To be able to basically mimic a data center setup of Hashi tools on your laptop, I think is incredible. And that it pretty much ports to your data center setup afterwards is super incredible and has saved my ass so many times, especially in my previous job when I was working with a Hashi stack.So it was such a great way to learn how to use it, to have a setup that could mimic what we would have in real life without me having to figure it out. I appreciate that you figured all that stuff out. If you could talk a little bit about HashiQube and what inspired you to start it, where it started. And now, what are some of the new capabilities?RIAAN: I totally hear your sentiment about being able to test something and mimic it in production because it's just so valuable. But really, where it started is when in South Africa, I was director of DevOps for Mage Mojo, a company that used to run Magento e-commerce stores on Kubernetes. But I really was looking for a visa, and I came to Australia and I was applying for so many jobs. I mean, if you can imagine applying from South Africa for remote jobs. I found it quite challenging at that stage, and I got a job as a consultant, and I was off the tools, mostly off the tools as the director of DevOps. But then being as a consultant, as you can imagine, your hands on the tools and that stage. I was working for Maritime Road Services. It's a government agency here in Australia and New South Wales.And I was subcontracting for a company called Wipro. And the stack we were working on was Jenkins as the CI/CD, Ansible Tower as the configuration management, getting secrets from Vault, and then Vault maintaining these secrets and everything orchestrated with Terraform. So Terraform would install Vault and Terraform Enterprise at that stage and maintain the stack. So at that stage I was living in the central coast and my train ride was about 1 hour, 50 minutes, 2 hours. And I was new to Vault and I was new to Terraform and I was just like, oh, I need to get this stuff in my head. But then as I go through the central coast, there's this river where there's no mobile connection and it was just difficult to get Internet and download stuff. So I thought, I know I must do something different. And Vagrant, I used Vagrant before for developer environments, vagrant.And then I put Vagrant with Vault and some Terraform in there with local stack so that I can learn how to code Terraform but not having a cloud account. And then when I get to work, I would try get access from Ansible Tower to this Vault and it just doesn't work. And I would go to the vault administrator and say, look, I think there's something wrong with this policy. And they were like, no, no, it's working. I was like, okay, well, now I'm going to test it on my local. I'm like, you see, if I remove this star, I don't get down the secrets, I don't get access to it, but if I add it, it works. So I used to go to the Vault guy and say, look here, this is my lab. This is where I'm testing it.I think the problem is here. And lo and behold, the problem was there. And since then, as a consultant, you work on Kubernetes with Helm. And then I would quickly need to test some Helm Charts or Docker builds and DBT with Airflow. And this is really where HashiQube started and I needed a place to store my configs and this is where HashiQube came about, where I could just text and store my configs and that's the start of it.ADRIANA: That is so cool. That's amazing. Yeah. And I can't say enough good things about HashiQube, because it's got all things. I want to go back to something that you said earlier. So you said that you used to be a director of DevOps and then when you moved to Australia, it sounds like you got into more hands-on stuff as a consultant. How was that transition like going from a director where you're not hands-on, to getting nitty gritty into the hands-on? How did that feel? What prompted the career pivot?RIAAN: First of all, it was insane. I was so overwhelmed, I had impostor syndrome on steroids. The people that I worked at that consultancy, Servian, were extremely professional, and even just the way they looked. And when I came to Australia, the accent was quite thick. So I would sit in a meeting and they would speak English, but I wouldn't understand a word. They would use abbreviations. And so I felt completely overwhelmed, but I would just be consistent. Look, you've hit some goals in the past.It's not like that. You don't know anything. But it was incredibly overwhelming because I used to use AWS and Cloudformation very successfully. Now, I don't know one line of Terraform and the Hashi stack with Vault, and it was just so overwhelming. But I must tell you, having a lab creates confidence. Having a place to test something out of the public eye, you can make stupid mistakes totally. It just gives you that place where you can figure something out and also break it slowly but surely. I decided, well, I don't know a line of Terraform yet, but I'm going to keep at this until I feel that I'm proficient and confident in Terraform.And I just kept at it. I started with the associate exam. I then started trying to give courses on Terraform. And then I became a Certified Terraform Instructor. I did my Vault Associate Exam. And then lately, I'm a Vault Implementation Partner, certified. And so, you know, it really starts off very organically. And so where I started and why I wanted to come to Australia is before that, I was for four years in Berlin, and my son was born in Berlin.But I really wanted him to know his parents and his grandparents and my brother and his kids. And you can't do that from the other side of the world. So we moved back to South Africa. You know, the situation there, I was retrenched four times in South Africa, and the place is a little bit, due to the corruption in government, there are quite high crime and murder rate, and you just feel unsafe. You have to look over your shoulder. As a man, you can handle it pretty easily. But my wife was always getting nightmares and stuff.And I just thought, like, I can't live like this, man. My kid is five years old. I need to give him a better future. I can always go back. I've still got some family there. But then I started looking around and as director of DevOps, my visa to the US didn't quite work out. It was dragging its feet. And so the guy said, well, you can go work in Ukraine with our Ukraine colleagues.So I had the visa stamped in my passport. But then this job from Australia came about and I was just like, oh, the language transition, the weather is more up my alley. Yeah, I'm just going to go for this. And I had the chance of staying director of DevOps, but I also had the chance of learning something new and doing something new. And I always kind of take, I wouldn't say the hard way out, but I take the uncommon, charted...that way. And so I'm so happy looking back at it, that I did come to Australia. That's the whole story. So now, hopefully by April, we'll be applying for Australian citizenship and that will conclude our five year journey.ADRIANA: Oh, wow.RIAAN: Citizenship in another country. I tell you what incredible last five years.ADRIANA: That is such an adventure. I mean, you're not only pivoting your job, but you're also moving to a totally different country, starting fresh. Like, so many changes, and just making it work.RIAAN: Yeah, I tell you, it was just absolutely incredible. But Australia is such a welcoming country. It's truly the rainbow nation with all of these nationalities. I mean, I go to my kids' school and I see Chinese and Filipinos and Indians there and know, and Kiwis from New Zealand and Africans and us from South Africa and all these kids play soccer together. And when I have my South African accent and the Indian parents have their accent, but all the kids sound Aussie. Yeah, mate. How are you doing, mate? And I thought always just. It is just so beautiful. I'm always astonished at how incredibly beautiful it is.ADRIANA: Yeah, that's so cool. Wow, that is such an awesome story. Thank you for sharing.RIAAN: My pleasure, my pleasure. It's such a feel-good story for me. I often look back at it and I'm just like, wow, it's so funny. Sometimes you look back at things you did two years ago and how this is now playing a role in your current day and age, but two years ago, you didn't know that what you were doing was actually going to, but you stick with it and you feed and it grows and. Yeah, that's so funny how life is.ADRIANA: Yeah, I totally know what you mean. I always tell people, everything that we've done in the past prepares us for this point in time, right in the present. And as you said, you don't necessarily know that it's going to lead you here. But it feels like it's been kind of in the works, right?RIAAN: Yes.ADRIANA: Or maybe because it happened.RIAAN: Yes. And if it feels good, do it. I liked your episode with Kelsey Hightower. I mean, he's also quite emotionally intelligent, and I would think quite a hyper aware individual to spot your podcast and ask the question. And, I mean, I really just am inspired by people. Like, I mean, well done, Kelsey. I mean, you've also inspired me. So hats off to you, mate.ADRIANA: Thank you.RIAAN: And I love your podcast and all the stuff you do. You're talking at HashiTalks now around the corner. Yeah, that's right.ADRIANA: HashiTalks. Yeah. And you've got a talk as well, right, for Hashi talks?RIAAN: Yes, I do. Everyone teaches you how to write Terraform code, but no one teaches you the scaffolding surrounding it, like dev containers, managing Terraform versions, scanning your code, doing the linting having environment, and everyone is like, oh, you must have micro repos. Mono repos is so bad. But this whole development lifecycle, just try to commit to three repositories with other maintainers and make prs and then wait and see how long you can get that code merged in. It is incredible. And so I'm going to give a talk a little bit about that to just help people get started and accelerate their Terraform development. So I'm looking quite forward to that.ADRIANA: Oh, that's awesome. That sounds like such a great topic.RIAAN: Yes. There's so much stuff that goes on behind the scenes that writing Terraform code is becoming the easy part.ADRIANA: Yeah, it's funny because I think, like many things, getting started withTterraform is easy. And then when you actually go to apply it for real life scenarios or know, I think a year ago, I was doing some work in terraform, and I want to clean up my code, and I'm like, I want to use modules. And I had everything working without using modules. And then I go to use modules, I'm like, crap, it's broken. You go to prettify your code, and it's like, another roadblock. But this is the cost of beautiful code. But these are the things that you don't realize as you go and evolve your code, right?RIAAN: Yes. And making your modules usable. So now you need to write modules and patterns. And I don't know if you've ever seen the Terraform EKS Blueprints repository. If you Google "Terraform EKS Blueprints", that is just such an amazing little project. So it's deploying EKS, but in there, they've got patterns and these patterns are just so well written. And if you look at the multitenancy with teams, one, I've used it at great success in my consulting gig last year.And I just want to say, hats off to those maintainers and developers. They've really done a good job. And if you ever want to see how to write...what good looks like, that would certainly be the repository to visit.ADRIANA: That's good to know. Thank you. Yeah, I just checked it out, as you mentioned, that it looks very well organized.RIAAN: It's incredibly well organized. It's really incredibly well...and when you start using it, you will see, oh, wow, there's been a great deal of thought that went into this thing.ADRIANA: Yeah, that's so cool. I always appreciate when folks put in that effort, especially in the open source world, because it's like extra work, right? And that someone was cared enough to just make it easily consumable for me is so nice.RIAAN: It's so selfless and I appreciate that little bit of it. I always think that people who contribute to open source projects, their glass is really overflowing because you have your personal life. I mean, you have kids and a family and a career, and yet you can still...and some people when they open up tickets, they're like, this doesn't work. Fix it. Yeah. Oh my goodness. Okay. And then you have to be nice. And I mean, it's really like helping a parent with their Internet problems. Right?ADRIANA: I know, right? Oh my God, so true. Yeah. And especially, as you said, the ticket is, "This doesn't work." And it's like, "Okay, can you tell me what isn't working?"RIAAN: Sounds so funny. But I always think back, our parents taught us how to tie our shoes and not to be cringey or anything, but they taught us how to wipe our bums. And really they had to have this insane amount of patience with us and try and try and try again. And I was trying to remind myself, especially when I've got a kid now nine years old, before that, I was kind of oblivious to the fact. But now that you've got a kid, sometimes you just have to stand back and laugh at the situation because it's just so funny. This development thing takes time.ADRIANA: Yeah, it's so true. That's a perfect way to describe it. Because when you have a kid, you're seeing your kid experience things for the first time, things that you take for granted, right? Like learning how to walk, learning how to crawl, or them, like when they're babies and they discovered that they have feet and they stick their feet in their mouths and you're like, oh, that is so cool, right? And these are things that you don't think about because it's like, yeah, I know where my feet are.RIAAN: I forgot what it feels like. Or what it tastes like to have your big toe in your mouth.ADRIANA: Right?RIAAN: I don't know what a big toe tastes like anymore.ADRIANA: Yeah.RIAAN: But I love the open source thing and also try to make things easy and consumable for people. I think that's the ultimate goal.ADRIANA: Yeah, absolutely. So much work goes into open source and I think I'm heavily involved with OpenTelemetry and I'm trying as a personal thing that I am trying to live by, which is like recently I was developing some content around OpenTelemetry and then I was going through the docs and realizing, oh, it's missing some stuff. And so I'd write a blog post about it to clarify it. But then I thought, well, that's nice, but it's missing stuff from the OpenTelemetry docs. Let's be a good citizen and contribute back to the OpenTelemetry docs, right? If there's something that you can contribute, even something so simple like documentation, clarifying documentation, I think it's so important if you're able to take the time and make that pull request to make somebody's life a little bit easier, right? Because oftentimes the developer docs for an open source project tend to be your first point...where...your one stop shop, hopefully...They're definitely your original landing point, right? So let's as a community try to make these docs better, right?RIAAN: 100% agreed. 100% agreed.ADRIANA: Now, I wanted to switch gears a little bit, but still on the Hashi train of thought, you are wearing a Core Contributor t-shirt for HashiCorp. I was wondering if you could explain what that's all about. Like what does a HashiCorp core contributor do and what led you to there?RIAAN: I got this last year in the post and I was just so happy to get mean. The Credley page says, "HashiCorp core contributors are individuals who are committed to the spirit of open source. They actively contribute to HashiCorp open source tools through submissions of pull request issues and bugs and contributor to documentation while advocating and adhering to the HashiCorp principles." And I've done a few pull requests and I help test stuff. I contribute to bugs and if anything, I just validate it and say, I've run this, I've tested this, it does work, whatever. I've got this problem here. And that got me this t-shirt, and I was just incredibly thankful. HashiCorp is quite a stunning community, and the individuals that make up this, I mean, you know, from the Ambassadors, they're a fun bunch.They...the you, they...the me, they...the other people in the community. And I do think that they've got a certain gravitas to attract these certain individuals, like looks for like, and I feel welcome there, and I like contributing there. And just because it's such a nice stack. I mean, Mitchell, Hashimoto and Armon Dadgar, they really made something really mean. I do know they went through this BUSL license change, but I mean, it was kind of expected, right? It's a company. It needs to make money. We live in a material world. We all need to make money.I understand it. To me, just the logical evolution of this next step. But that said, the contribution that they've made to open source and to helping people like me learn and the stuff they give us for free is just incredible. So I'll be forever thankful for that.ADRIANA: That's so cool. And I love that you're being rewarded for your contributions with this designation. I think it's so awesome.RIAAN: I do appreciate it as well. I contributed such a small contribution, and still they recognized that, and I was just thankful and appreciative. It's beautiful. It feels good to get a little gift or something.ADRIANA: Yeah, totally. It's nice to know that the community appreciates. And on the same vein, like you mentioned, you and I are both HashiCorp Ambassadors. And actually you're the one who nominated me initially for the HashiCorp Ambassadorship. So I definitely appreciate that.RIAAN: You know, because I always say, like, I meet a lot of people in my work, and this is not to be bashful or anything, but a lot of people are...If you can imagine a heart monitor and you see a blip on that monitor and I see blips, and I think that those blips should be recognized and called out. I think we should be the type of person that say, wow, you look good today, or, this is inspirational. I read your blog post, and I was actually surprised when I saw that you wrote all of these blog posts using HashiQube. I was like, wow, this thing has been out in the wild. And this is the first time I see it, and I was blown away. I contacted you, I think, over Medium.ADRIANA: Yeah, that's right.RIAAN: Because you were not only using HashiQube, but also writing about it and using it in different ways. And I was incredible. And that's exactly why I nominated you because I think these type of people should be called out and should be celebrated. And you were certainly inspiring to me. And if you were inspiring to me, I bet you you're inspiring to many others out there. And that's the next wave of Ambassadors coming up in the world.ADRIANA: Yeah, for sure. It's been a great program so far. I think I've been an Ambassador for two years. When did you become an Ambassador?RIAAN: 2021.ADRIANA: Oh, awesome.RIAAN: So this year I'll be an Ambassador again. You have to put in the work and the street cred and stay active in the community and stuff. But while you can, you should. If you can.ADRIANA: Yeah, absolutely. I put in an application again for this year, so fingers crossed I get it again. Fingers crossed. Yeah, it's been great through the Ambassadorship program. They invited me last fall to MC HashiTalks Deploy in December. So that was fun. That was so fun.I'd never MCed before, so I was super nervous. But they were very organized. They're like, this is how it's going to go and this is the order. And here's a table of who the speaker is. You just need to fill out this stuff as prepare a script for yourself. So it was like, okay, because I was full on panicking when I agreed to become an MC. I'm like, okay, that'll be easy. And then there was like all this process.I'm like, oh my God. It is very overwhelming.RIAAN: But they do make it easy for you, but they do support you in pulling it off. Easy is definitely the wrong choice of words, but they do very much support you in getting this thing across the line. And then in the end you look back at it and you're like, wow, that was fun.ADRIANA: Yeah, it was a great experience and I'm so grateful for the opportunity that I was afforded because of being an Ambassador. So it's nice to have these little things here and there.RIAAN: I love it.ADRIANA: Now, one thing that I wanted to ask...you're very involved in...you do a lot of Terraform work. Have you played around with the now competitor OpenTofu?RIAAN: That's a good question. And no, I have not. I mean I did use Terragrunt before and I actually quite like Terragrunt. And to be honest with you, I don't think that that was nice to make OpenTofu. I'm an open source guy, man. I've been using Ubuntu Linux since 2008 and I started using RedHat in 2000, actually RedHat 6.2. And there's always a way to go about things. And I believe in having diplomacy. Someone created it.And now you're kind of like taking ownership of this and you're taking it. And that's also against the spirit of open source. So I have not tried using OpenTofu. I actually cringe when I hear that name. Sorry to say it, I know what they did with OpenTofu. I mean, I did think about it. It's Open TF Tofu and whatever, but I won't be using it. I'm just so know, it just feels weird to me.It just feels wrong to me. And so I like Terraform. And in the same breath, I also haven't tried Bicep from Azure. I'm a ashicle guy, I'm a terraform guy. So I have not delved into that. And I wish them luck on their journey and stuff. But when I see that name, it's just worthy to me. So I've unfortunately not tested it out or anything.ADRIANA: Yeah, that's fair. That's fair. Now, going back to one thing that you mentioned earlier, which was Terragrunt. Can you explain to folks who aren't familiar with Terragrunt what that's all about?RIAAN: I mean, I do like Terragrunt. Just touching on the topic. I wish that they could have just played nice because it could have really benefited this ecosystem so much more. And the companies...there is enough money in the world for everyone. Trust me, there's no reason why...there's enough money. There's billions and trillions and gazillions. So there's always an amicable way to do something.But getting back to Terragrunt is very. I like what the Yevgeni Smirnoff did. You write your module? Everything driven through variables. And so your module should be completely flexible, very dynamic. And then what Terragrant is, is they add a Terragrunt HCL file and then you can make your folders names, variables. So you can imagine if you've got an environments folder and you've got Dev, Prod, UAT, Low, Production, whatever, Non-prod in there, you can turn this folder name into a variable. So you can then define this thing at the very top-level and benefit from this in your modules. So you can say module name and you can use module name in your tags.So when you do apply this Terragrunt stack, these Terraforms, you can benefit from all of this modules that you define in the top and top down. So for those who's ever used Puppet and Hira, it's very similar to Hira. So in Hira, you've got a common file and this common file can be used down in your hierarchy. But let's say you want to overwrite a key name down in a couple of folders, let's say environment. Then you've got dev, and then in dev you've got your availability zones or your regions and then further down you've got availability zones that you stack support. And then lastly you've got your Terraform module and you just want to override a key on a module somewhere down you just overwrite this key. And so what, Terragrunt was quite nice as they defined everything in YAML, so you can have complete very complex YAML code structures that you can then pass to many, very many Terraform modules.And these things all get executed in parallel. And so you can bring up complex infrastructure environments quite quickly. And because your code is DRY, your Terraform modules can be used many times over and you just pass parameters to it which is defined in your YAML files. And this is how Terragrunt comes about. It's actually beautiful the way they've done it. It's really nice. It becomes a little bit complex when you debug yourself because if you can imagine you've got ten Terraform threads now running all at once and if one breaks, the rest of them also stops and it's like quite an avalanche of output. But as far as if you get to use it and you use it properly, then you can accomplish quite a lot very quickly.ADRIANA: Cool. And on a similar vein, maybe not so much Terragrunt, but in general for Terraform, how do you test Terraform code?RIAAN: So my Terraform code, what I do is I have an examples directory or a patterns directory next to my modules. So if you can imagine I would have in my top gun Terraform developer environment I would have Terraform and then AWS, GCP and Azure and custom. And inside those I'd have modules folders and inside of those I'd have our Terraform modules. Then next to the modules folders I would have patterns and the pattern would be Linux server behind load balancer. And that Linux server behind load balancer would just be a main and a variables and outputs that then reference these modules with the source stanza inside of these modules. And then I just build them or I run them and I apply them. I normally just do a plan and I see if it works. But I do run them through an init and if I want to test it all at once, I actually drop a Terragrunt HCL file in there and I use "terragrant run" or "plan" to test all of these things.I use Terraform in conjunction with this and then I plan all of these modules quite quickly. And if my plan works, I leave it out there and then I wait till I run into it again or someone needs an update or something. And then I look at this again.ADRIANA: Cool, that's so awesome. Well, thanks for sharing. We are just about at time, but before we wrap up, I actually have two questions. One, what is your favorite HashiCorp tool?RIAAN: My favorite HashiCorp tool would really be Terraform at the moment. There's a few. There's Vagrant. I love vagrant.ADRIANA: Vagrant is great. I really love it. It was my first Hashi tool.RIAAN: It's incredibly powerful. I mean, I really must take a shout out to vagrant. I mean, thank you, Mitchell and Armon for writing this thing. I use it every day, still. It's incredibly powerful. So I love Vagrant. I dig Terraform because that's my staple. I eat that thing every day for breakfast.I love Nomad. I run Nomad jobs quite a lot. And so nomad is just so easy. You just drop it on a server and there could be still PHP and Apache sites running on there, but there's Nomad with containerized jobs and you can just migrate it and it's so cost effective and so easy to test it. And I've also liked Waypoint at the moment.ADRIANA: Oh, Waypoint, yeah, I haven't played with Waypoint for a while. Yeah, I need to play with it. Because I think when I played with Waypoint, it was very early days and I can early days. I'm so curious to see how it's evolved since then.RIAAN: It's got a lot of potential, and then Boundary is the next thing I really need to sink my teeth in and get a couple of examples into HashiQube. Just get people started and that's on my to do list to do. But yeah, there are so many.ADRIANA: So many awesome tools.RIAAN: You know what I mean? To pick a favorite. I mean, it's even difficult to pick a favorite cloud because all of these things just enable you to do stuff. So mean. GCP has got its way of working and Azure has got its way of working and AWS works in its ways, but they all help me on my day to day and I'm just so thankful we've got cloud computing. I mean, holy moly, can you imagine? Still back in the day.ADRIANA: I know, right? Yeah, it's wild to see how much software has evolved in the last 20 years. Holy cow. Mind blowing.RIAAN: Mind blowing coming from NT4 and A+ where I started with chips and RAM and stuff. I mean, it's incredible to see how it's evolved.ADRIANA: I totally agree. I totally agree. I mean, there was no cloud when I started my career.RIAAN: No, just think back fondly. I mean, I used to use Gentoo and compiling stuff and running my own postfix mail servers and pure FTP servers and. Oh my goodness. Incredible.ADRIANA: And now look, the world is at our fingertips with cloud. That's pretty mind blowing. Well, before we wrap up, do you have any final words of wisdom for our audience?RIAAN: Well, maybe if you want to check out hashicube. I always plug that little thing. It's just so incredible to see a little docker container running more docker containers.ADRIANA: Oh my God, it's like mind blowing sometimes.RIAAN: Just think back and how lucky I was to get that to work. It is just incredible. And so easy to POC stuff and get stuff up. So, I mean, if you want to check out HashiQube, if you want to learn or play around with, that's my DevOps lab from now on going forward. Yeah, so cool.ADRIANA: It's a great lab.RIAAN: And that's the only plug. And see you guys at HashiTalks in a couple of days.ADRIANA: Yeah, totally. The other thing I want to mention on that same vein is I think you getting vVgrant to work with the Docker provider is probably one of the best running examples of Vagrant with the Docker provider, because I don't think there's a lot of documentation around that. So thank you for that. Hats off to you because, yeah, I think getting that to work, which you did, to be able to run HashiQube on the M processor, Macs, that's why you needed to get that running, right.RIAAN: I so like it because it is just so light and if you do Vagrant SSH, it's very difficult to say you're in a Docker container now.ADRIANA: Yeah, I know. You would never know. You would never know.RIAAN: And it's incredible. I can really see things going that way. It's the way I do stuff at the moment. I no longer do VMS, so even when I run HashiQube on an EC2, or when I want to run Ansible AWX Tower on an ec two, I just HashiQube and "vagrant up".ADRIANA: Yeah, it's the way to do it. I love it. Well, thank you so much.RIAAN: Thank you for having me on your show. It's so good to see you. And shout out to your daughter, who I believe is doing your editing for your videos and job well done. I take my hat off. Thank you so much for your time and it's so good to see you again.ADRIANA: Yeah, it was great to see you as well. And thank you, Riaan, for geeking out with me today. And y'all, don't forget to subscribe. Be sure to check the show notes for additional resources and to connect with us and our guests on social media. Until next time...RIAAN: Peace out and geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking out is also produced by my daughter, Hannah Maxwell, who incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going going to bento.me/geekingout.

About our guest:Edith is a Tech Evangelist at Percona, a company known for its work with open source databases. She used to work as a DevOps engineer, helping IT companies and startups set up and use DevOps. After taking a break for two years, Edith started working with Open Source, which helped her get back into the job market. She has made valuable contributions to the Apache Airflow project during her time with Outreachy and is working on translating the Kubernetes website into Spanish. Edith is also an ambassador for the Cloud Native Computing Foundation, focusing on creating content, and is recognized as a Docker captain. She has taken part in tech programs like Stanford's Code in Place and studied at 42, a coding school in California. Recently, Edith moved to the United Kingdom on a Global Talent Visa, which was a big step forward in her life.Find our guest on:X (Twitter)LinkedInYouTubeFind us on:All of our social channels are on bento.me/geekingoutAll of Adriana's social channels are on bento.me/adrianamvillelaShow Links:Cloud Native Computing Foundation (CNCF)OutreachyApache AirflowKubernetes Community Days (KCD)Liz RiceKCD Peru - July 20th, 2024KubeHuddle Toronto 2024Additional Links:Docker Captains programCode in Place (Stanford University)42 Silicon Valley (coding school)Transcript:ADRIANA: Hey, y'all, welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. Geeking out with me today. I have Edith Puklia. And where are you calling from today?EDITH: Yeah, I am calling from UK. London, UK.ADRIANA: Awesome. I've had a few people on the show that have called in from London. I think you're like the third person from London. I had Abby Bangser, who I think it was Abby who introduced. Right? Abby is the ultimate connector of people. So thank you, Abby, for introducing us.Yes, I had Abby and then Jennifer Riggins, who is a tech journalist. You probably saw a bunch of her pieces on The New Stack. And then you. So you are my three London, UK people. Very exciting.EDITH: Thank you.ADRIANA: And we share a South American connection as well, right?EDITH: Yes. You are from Brazil, right? Peru here.ADRIANA: Yay. Home of the llamas.EDITH: We love llamas. We love them.ADRIANA: Oh, yeah, you have the awesome mug. Yeah. I was telling you earlier before we started recording that llamas and capybaras are like my two favorite animals in the world, so I always get excited when I see either one of them. Cool. Well, let's start with the lightning round questions. Are you ready?EDITH: Yes.ADRIANA: Okay. Are you left-handed or right-handed?EDITH: Right.ADRIANA: Okay. Do you prefer iPhone or Android?EDITH: iPhone.ADRIANA: Okay. Do you prefer to use Mac, Linux or Windows?EDITH: Linux. I love Linux.ADRIANA: All right. Hardcore. I love it. What is your favorite programming language?EDITH: Okay, there are many. Now my favorite right now I can say that it's Rust.ADRIANA: Very cool, very cool. I hear that it's great. But also very complicated to get into.EDITH: Yes. I mean, I don't code like a deep programming. I am just starting, just learning, but I was fascinated for what you can do with it.ADRIANA: Cool. I'm curious as a sidebar, what got you interested in learning Rust?EDITH: Because how you can easily integrate with other technologies. For example, with Docker I was trying to play, I was able to do fast with Rust. And using Chat, GPT is also a great tool to learn,.ADRIANA: Yeah, yeah, that's awesome. Very cool. Okay, next question. Do you prefer Dev or Ops?EDITH: Hard question here. Yeah, I prefer Ops.ADRIANA: Okay, cool. Next one. Do you like JSON or YAML better?EDITH: YAML. I feel that I can read it.ADRIANA: Yes. Yeah, that's my thing with YAML too. I think it's easier to read. Okay, next one may be controversial spaces or tabs? Which one do you like better?EDITH: Spaces or tabs? I use spaces.ADRIANA: All right.EDITH: Yeah. You?ADRIANA: Okay. So I used to be a big fan of tabs, but then I started using spaces, especially when working with YAML, because it felt a little bit more organic for me. Yeah. So I used to be very adamant, like, no, it's got to be tabs. But now I'm like, I'm open right now. I'm down for spaces. So, yeah.EDITH: Okay.ADRIANA: Also, kudos to you for turning the question back on me.EDITH: But I am curious about you too. Why you too?ADRIANA: I love it. Very awesome. Okay, two more questions. Do you prefer to consume content through video or text?EDITH: Okay. I love videos. I have a hard moment reading a lot of text, but videos is more easy for me to consume for you. I can imagine that too, because you do videos a lot also, right?ADRIANA: No, it's mostly text for me. It's funny, though. I was talking to my dad yesterday, so my dad does not...he was like, I do not like podcasts. I'm like, but my podcast is on video, too. He's like, it's just boring to see people's heads on video, but he's more of a video guy because he likes the visual stuff. He refuses to do podcasts. And my daughter loves, loves, loves videos. She's always learning things on Instagram or YouTube.EDITH: And you have a lot of articles.ADRIANA: Definitely. Like, I prefer writing. I think I've embraced video a little bit more. I used to be very scared of editing video, and I feel like nowadays the tools have made it easier to do video edits so that it looks like I'm not fumbling around. So I feel a lot more comfortable doing video editing compared to, like, ten years ago when it felt impossibly hard.EDITH: With writing. I feel really hard writing. Long time ago, I was not able to write a single article that take me too long to write. But now I feel I'm more comfortable because I am trying to do constantly.ADRIANA: Oh, that's awesome.EDITH: Yay.ADRIANA: I love to hear stuff like that. Final question. What is your superpower?EDITH: Patience.ADRIANA: Patience. I love it.EDITH: Yeah. You?ADRIANA: Oh, jeez. My superpower. I think I'm really good at connecting people together. I find myself in situations where I'll have a conversation with someone and then they'll ask me a question. I'm like, I know a person that you can talk to. Yeah.EDITH: You have a lot of people in your mind.ADRIANA: Yeah, I guess so. I guess so. At least remember people who should be talking to each other.EDITH: That's a superpower.ADRIANA: All right, cool. Well, that was it for the lightning round questions. You survived! Yay.EDITH: Thank you.ADRIANA: Okay, so now for the fun stuff. As I mentioned before, we got connected through Abby, and then it turns out we have another connection in common, which is we're both CNCF Ambassadors from the spring 2023 group. So, very exciting. I guess our first year of ambassadorship is coming to a close, and I guess they're renewing applications end of this month. So my question to you is, how has it been this last almost year as a CNCF Ambassador?EDITH: Almost a year because we started at March. I think the last year. I was here in London, too. Then I go back to Peru. And how I feel this year being CNCF Ambassador, I think it doesn't cost to me too much make things for being Ambassador because I was in the category. If you see there are several categories, right? Run events or you go many, you can choose whatever you want. I choose the part of content creations which I love. So when I inspire it, I just create a video. I just make a flyer or a pdf of anything which I do in my free time. And I love it because editing videos and making that things require a lot of patience.ADRIANA: Yes. There's your superpower.EDITH: That's my superpower. And I can do that. I feel really excited. I feel like I'm going to apply again. For the last month, I was not just involving in content creation, I was also involving in organizing events. We are organizing Kubernetes Community Days. Lima, Peru is the first time we are running these events in Peru with other members of the community and also being members of CFP proposals reviewers, for example. I was involved in many other things.No just content creation. A lot of things to learn. A lot of things that I never did in the past, but I never thought to do it. But I am doing. Wow, this is amazing. It's hard sometimes because it costs to learn, but it's very interesting and I like.ADRIANA: Yeah, yeah, I totally agree. And I have to say, I really enjoyed being a CNCF Ambassador because of the different opportunities that it's opened up, like just making new connections and being given opportunities to review CFPs and being given speaking opportunities that you necessarily wouldn't have had otherwise.EDITH: Yeah. I feel in the same way, just to tell you that the first trip that I did in my life outside Peru was for CNCF because I won a scholarship. So I didn't speak English, just my name. And I got to Seattle and saw a different experience. Just being in the KubeCon in Seattle, it was just amazing. And things that made me think, wow, there is doors here that I should start open. It's here I should go. I saw a lot of opportunities, and since then I go to that side of CNCF and all those communities my career start to doing that. I think the support for women in tech is also very valuable what we are doing as a community.ADRIANA: Yeah, yeah, absolutely. I do want to go back to your earlier comment on your first trip out of Peru, and you said you didn't know English at the time. How long ago was that?EDITH: I'm sorry? It was 2018. Yes. I mean, I study English. Yes, I talk basic English, but outside you is different.ADRIANA: It's different, yeah. It's so true. Because it's the slang, it's the technical terms. It's funny, because I was thinking back...as you mentioned, I'm from Brazil, but I grew up most of...I've been in Canada since I was ten. I've been in Canada for, like, almost 35 years. So I am bilingual. I'm even trilingual.I speak French, too, but I have to say my Portuguese has degraded in the time that I've been here, even though I speak to my parents in Portuguese, but I lack some of the technical terminology and I even lack some of the slang. So I actually started joining...following people on Instagram for Portuguese language school so that I can up my game to just get back into some of the slang terms and just be a little bit more conversational than I am, because I've lost some of that from not being around that many Portuguese speakers.EDITH: Yeah, I understand that. I have been here speaking English not too long time, but I already start to forgetting how to write things in Spanish, and I brought it wrong. And my father is always correcting me asterisk.ADRIANA: I know my dad's always correcting me as well, because sometimes I'll do a translation...what seems to be a direct translation of the English word to Portuguese, and he's like, yeah, that's not the same word. It means something totally different. I'm like, oh, my God, I feel so embarrassed.EDITH: You are not alone.ADRIANA: But then I remember something that I've read, like, being able to speak more than one language and making the effort to converse in more than one language is putting yourself out there. It's a sign of bravery, because, holy crap, it is so scary to attempt to communicate it in a language that you're not necessarily familiar with or super comfortable speaking in. Before we met today to record this, I recorded a podcast episode in Portuguese, and it was my second time recording a podcast episode in Portuguese. And I was so scared because I'm like, I don't know technical terminology in Portuguese. And so some of the advice that I got from a few of my Brazilian friends who live here in Toronto, they're like, "Don't worry if you don't know the word. Just use the English word, but give it a bit of a Portuguese accent." Yeah. I mean, like, you know, even though, like, something like that completely scared the shit out of me. At the same time, I'm like, you know what? I'm going to force myself to do this because the more I do it, the more comfortable I will get.EDITH: Yes. I don't know why we are like that. I mean, we are really afraid. We jump and we start to doing. Then it pass and we said we did it. Yeah. Before that start to feel like the fear, the hands start to with everything, that scary moment. Then you use go, but then you jump to another thing.EDITH: To start to jump to a ring and another ring. The same motions.ADRIANA: Exactly. It actually reminds me of, like, I was having this conversation last week with someone where I'm like, oh, my God. When I first learned about cloud and cloud native, I'm like, it's this terrifying, scary thing. So I was like, I don't want to do it. I don't know. I don't think I can do it. And then I did my first thing in the cloud and I'm like, oh, okay. It was okay.ADRIANA: Yeah, it wasn't scary.EDITH: You are complete. Nothing happened. It's weird how we can be afraid of things that also involve human beings, like communications, like speaking, we are afraid. I don't know what we are afraid. What is the fear that we feel to be exposed, to see that others look at us and we are trying to embarrass. I think we all are humans and we all have the mistakes.ADRIANA: Yeah. And I think we judge ourselves a lot more than others judge us. When I'm having a conversation with someone in Portuguese, especially like, with my family in Brazil, and thankfully know Google translate to help me when I'm on WhatsApp, but I'm like, oh, my God, they're going to look at me and they're going to make fun of my grammar, whatever, or use the wrong word. But then I also have to remind myself they have better things to do than to nit-pick on your grammar. They have their own lives. Get over yourself. It's not all about you.EDITH: Yeah.ADRIANA: Okay, so I want to switch gears again and talk a little bit about your career, like how you got into...and I know you do a lot of work around Kubernetes and containers. What got you in it?EDITH: Yeah. Okay. I was in the field of tech for almost ten years. I can say I work it as a DevOps, also as a developer for big companies in Peru. For companies where I started from scratch, things. Was really hard. For example, when DevOps was not big tendency. Right now we are starting from scratch. I started from scratch alone.Trying to start servers, make all that stuff was really hard, but challenge. And after that I decided to quit my job in 2018, I think...2019. Because of healthy problems, emotional problems, healthy problems, back problems, and with family problems, everything like when you have one and everything start to make a big thing. And I decided to take a moment. I take two years. I never thought it will take me too much, but I took two years. Okay. But these two years was really amazing for me. It was amazing because I give me this time to know me better.Things that I never did in the past. Because I was always running, running, piecing the car. I don't know how to say the accelerator of the car and trying to gas in that life. But then when everything happened, I just. No plan. Nothing for that future, for the future, just that. Just myself, my thoughts and my body. And thinking what made me happy, what will make me happy for the future.It's how I invest the time in two years. So not just thinking, but also doing. Because I wanted to improve English, I wanted to improve also my technical skills. And I realized that tech made me happy. It's one of the things also make me happy. Okay. I'm also geek.ADRIANA: I love it.EDITH: Yeah. Between several things, tech also made me happy. And I start to improve my skills. I start to learning English, which was really challenged for me. Now I can communicate how I want. I think I need to improve, but it's good for me. So I started to apply for jobs after having an internship in Outreachy. Did you hear about Outreachy?ADRIANA: Yes, yes. I have heard of Outreachy. For folks who have not heard of Outreachy...EDITH: Yes. Outreachy is a program, open source program. In three months you can have a mentor. It is also paid. So you learn a lot of things because you put your hands in real open source projects. I put my hands in Apache Airflow, where I start to code. I start to make things that I had never thought to do it. It was really amazing. And I wasn't with Oyo, but they give me a pay.So it was enough to me to survive and to learn English and improve some soft skills and also technical skills. Then I started to apply a job. I set a goal for me, for myself, to apply for an international company where I can speak English. Have that opportunity to speak English. So applied maybe to 200 jobs in two months. I applied the most I can.ADRIANA: Wow.EDITH: Sweden, Germany, USA. I send my CVs a lot. So one of the companies was Percona, and after the process and everything, I was hired by Percona and now I'm working as a technology evangelist in Percona, which is an open source company.ADRIANA: That's so cool. And I have to say, it so resonated with me when you said that as part of your time of really digging into who you are and what you love, that you decided that you love tech. Because I felt like I went through a similar thing in my career as well. I was working at a bank and I had quit my job at the bank to become a professional full time photographer. And I was like, this is it. I'm done. I don't want to work in tech. I want to do photography.This is my passion. And I did it for a year, and then I came to this moment in my life where I was like, so it's really hard. And if I really want to make this work, I can probably give it another year or two and probably finally start seeing growth. Because at the time, it was like I wasn't really right then I thought, but do I want to invest this extra time to grow my photography business? What do I actually like doing? Then I realized I had more fun tinkering around, like doing my newsletters and tinkering around with my website, and I was using WordPress and I bought this plugin that wasn't working. And I'm like, let's go into the PHP code to see what's wrong. And I'm like, oh, I think I like that more. So I ended up like. I'm like, you know what? I want to go back to tech.It took a year of me not being in tech to realize that I actually enjoyed tech. So, anyway, yeah, your story so much resonated with me, and I think it's so awesome and so important to take the time in our careers to figure out what makes us happy because, I don't know, we're at work for most of our lives and it better be something that we enjoy, right?EDITH: Yeah. And it's different and it's unique history. I can say yours, for example, is totally different than mine, but it's very unique. It has the meaning for you, and that is the good thing, the very important thing that maybe nobody's going to understand. But we are going to understand, right?ADRIANA: Yeah.EDITH: We are the only one who understand that special moment. Yes.ADRIANA: It's so true. Very deep thoughts. I love it. So philosophical. So great. The other thing I wanted to ask you about, because how did you get into doing Kubernetes work?EDITH: Kubernetes in Peru, we started to hear about Docker, for example, we started to see the whale everywhere. What is the doll? And that was the curiosity, the doll with a terminal. The terminal to Docker and containers and all that stuff. The same, I think happened with Kubernetes. In Peru, we started to listen about Kubernetes as a technology, as a standard things, but just listening. So I started to follow. What is this Kubernetes thing that people talk? And I start to follow people on social media, like Liz Rice, for example. The first person, people that I was following was Kelsey [Hightower].You interview Kelsey. Kelsey, Liz. That's big people there. So I was really fascinated for the keynote that they did. So I started to investigate about KubeCon, and it's how I got the scholarship to go. And once I go there, I see, wow, Kubernetes is the thing. So I started doing some demos at home with Google Cloud because I had free credits. So I start to play because it's what I want to. I love to do, to play with technology.Do it, destroy it, create it, destroy it. Mac, Linux, Windows, destroy it again. I don't know. But I found it funny. Funny, yes. Funny. Enjoyable? What is the word? Okay.ADRIANA: Yeah. Fun.EDITH: Yes.ADRIANA: That's awesome. It's so cool. And I especially love what you said about creating and destroying. And I think that's honestly some of the most fun stuff about playing with Kubernetes clusters is like, you do a bunch of stuff, you mess it up. Okay, time to start over again.EDITH: We are not in production, so you can destroy.ADRIANA: Right, exactly. Yeah, definitely don't do that with your production cluster. So you mentioned playing around with Google Cloud. Have you played around with any other cloud providers?EDITH: Yes, I had the opportunity to play with Red Hat, with Amazon, with Google Cloud and Azure. Yeah.ADRIANA: Cool. And which one's your favorite of the ones you've played with?EDITH: Google is my favorite. I don't know. I feel like interface, the graphical user interface, was more, for me, easy to do it, easy to create, to understand. For me, I feel that in that time, I feel that Amazon has a lot of things. Maybe that I didn't get too much distracted. But anyway, I use in the same time the three cloud providers.ADRIANA: Cool. That's awesome. So switching gears a bit, I wanted to talk a little bit about some of your community work.EDITH: Thank you for that question and community. I started in community participating as all of us just going to the events and see people talking and watch. But then I say, okay, there is another weight because you are in a certain level I can say you advance a little bit in your career and you say, okay, there is people who did a lot for you. They give time, they prepare. So you learn. So let's make it something for that too. And it's the mindset of the community, right? Get back this kind of things. So now we started with creating communities.I say we because it's not just me, we always work with people in communities and we created communities in the city where I was living. There was where I was living lacks of community techs. There is no much communities in tech. So it's where I wanted to start. I'm going to create communities with many people. So Docker was one of the companies that helped me to make it with sponsoring some events. So we start to create events for per year. For example, we celebrate the anniversary of Docker. Like, the 10th anniversary which was a lot of people going to that event and they are learning about the technology and that is one of the work that we are working until now.I like of that and I feel proud about that because we are doing something small but maybe could be impactful and give this opportunity to people that don't have the opportunity to make in that city without leaving the city. Yeah, this is one of the things that we are doing and the other is CNCF. I love this. So we had this big opportunity also because CNCF sponsor it. We have all the support of CNCF to make it possible a Kubernetes Community Days in my country in Peru. So we as a team because we are several people working in that we are creating this community for this year, for July. So I hope we can see it and we can repeat it over the year. So this will be impact also and generate more opportunities for people in our country.ADRIANA: That's amazing. Now how much work goes into putting together a Kubernetes Community Day> But actually before I get you to answer that, maybe it would be helpful to explain to our audience what is a Kubernetes Community Day? What's the purpose of having something like that?EDITH: Yeah, these are spaces where we give people the opportunity to share about the expertise they have about the Kubernetes and the CNCF ecosystem that exists. So a Kubernetes community days is an event. Could be in person, online or both, two days or one day. We choose that. And where several experts or people who want to share about ecosystem of Kubernetes go and start to talk about that. Could be not just talk, could be workshop, could be several things lightning talks, open forum, things like that. And sometimes it's free, sometimes it requires some payment. It depends on the organization, but it's a big opportunity to join a lot of experts, beginners, enthusiasts, members of communities between all this ecosystem. Kubernetes ecosystem.ADRIANA: That's amazing. So it's basically like a little mini conference.EDITH: Yeah.ADRIANA: It mini though? It sounds like. It sounds like a lot of work.EDITH: We compare it with KubeCon, could be mini, but to be honest, it's not like to be mini. Not mini like I saw 500 people in some of the Kubernetes Community Days in Europe, I think.ADRIANA: Holy cow. Damn.EDITH: We are targeting in Peru for the Kubernetes Community Days in Peru, we are targeting also 500 people. Yeah. Attendees.ADRIANA: Amazing. That's so cool. And so for organizing Kubernetes Community Day or KCD, what type of support do you get from the CNCF? As a CNCF Ambassador I would imagine that you get a little extra boost of support from the CNCF? So if you could talk a little bit about that?EDITH: Yeah. What we have is support from members of the CNCF, people who work there. So they help us organize and we have synchronization meetings sometimes to see how is our progress. Also they try to support us the most they can. For example providing us the logos and designer people who can also help us. They also sign a budget for coupons, courses, coupons and some budget. I don't remember the amount of the budget to start the event. That will help us to pay some things and what more? I'm not sure about that but they give the opportunity also to travel to the KubeCon I think.But maybe I am wrong. I'm not sure about that but I think there is many opportunities. Once you are in the ecosystem and once you are doing things there are many opportunities. Networking is also a big opportunity because in an event you can contact with several people who also are organizing. This is my first time organizing so I don't have precise response how much that will take me because it's the first time that I am running it. Let's see how it goes.ADRIANA: So does the CNCF provide then the overall funding for running a KCD or do they provide some funding? Do you need sponsorships? How does that work?EDITH: Yeah, we need a sponsorship. Each team tried to find a sponsorship in the country or outside the country. So with that budget is how they estimate how many attendees we will have and how we are going to assign it. In some cases, this is free and the budget that you need is maybe less, right? It depends, to be honest, of the country and of the city of the country, because the governance community is now is for city. So let's give the opportunity to have more in a country.ADRIANA: Cool. That's awesome.EDITH: Did you think to organize an event? Did you think to participate?ADRIANA: So I'm actually helping to organize an event in Toronto called KubeHuddle, which is like...I think the first KubeHuddle took place in the UK, I want to say a few years ago. And then there was a KubeHuddle in Toronto last year that I attended as a speaker. So then the organizer of KubeHuddle, Marino, he asked me at the end of last one, he's like, "Do you want to help organize the 2024 one? I'm like, okay, yeah." So I am involved in that...because I have so many things on my plate, like, I'm trying to take on what I can without being overwhelmed, but still making sure that I help out. So this is my first experience with that. And KubeHuddle is taking place on May the 7th in Toronto. So this year it's going to be a one-day conference. Last year it was a two-day conference. This year it's a one day single-track conference. So yeah, very exciting. So is KCD Peru? Is it a one-day or two-day conference?EDITH: One-day conference.ADRIANA: One day. And is it multiple tracks or is it single track?EDITH: Multiple. We are thinking multiple.ADRIANA: Okay, cool. Awesome. Very exciting. I'm super stoked for you. I hope it all goes well. Now we are coming up on time, but before we finish off, do you have any parting words of wisdom for our audience?EDITH: If I can say something, it's enjoy life.ADRIANA: I love that. That is perfect.EDITH: See the sun. Look at that and enjoy it. It's very nice. Sometimes. If you have sun.ADRIANA: Except on cold days.EDITH: It's really cold. There is no sun.ADRIANA: I don't know...What's the temperature like in London today, because here it's a warm -4C.EDITH: Today there was a sun, but once you put the finger outside, it freezes. But the sun was lining.ADRIANA: That makes it better. Yesterday it was like -15C in Toronto and I went for a walk and I had to go into different stores to warm up. So I didn't freeze. But yes, I absolutely love your parting words of wisdom. I think we get so caught up in our work lives that we forget to also just take a break, reset, enjoy life. Enjoy the non-work time. Well, this was awesome. Thank you so much, Edith, for geeking out with me today. Y'all don't forget to subscribe and be sure to check the show notes for additional resources and to connect with us and our guests on social media. Until next time...EDITH: Peace out, and geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Vilella. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:Jennifer Riggins is a culture side of tech storyteller, journalist, writer, and event and podcast host, helping to share the stories where culture and technology collide and to translate the impact of the tech we are building. She has been a working writer since 2003, and is currently based in London.Find our guest on:X (Twitter)LinkedInMastodonBlueskyFind us on:All of our social channels are on bento.me/geekingoutAll of Adriana's social channels are on bento.me/adrianamvillelaShow Links:The New StackLeadDevAt QCon: Why Generative AI Is Harmful to Earth and SocietyE-lanceKelsey Hightower at Civo NavigateAbby BangserDiversity, Equity, and Inclusion (DEI)David Heinemeier Hansson (DHH)37signals (formerly Basecamp)GitHub CopilotBackstage (CNCF Project donated by Spotify)Divya Mohan (Kubernetes Maintainer)Octoverse: The state of open source and rise of AI in 2023The AI Revolution Is Just Getting Started: Leslie Miley Bids Us to Act Now against Its Bias and CO2Consequence Scanning – an agile event for responsible teamsAdditional Links:Developer Empowerment Via Platform Engineering, Self-Service Toolingtag-environmental-sustainability Slack Channel (CNCF Slack)Transcript:ADRIANA: Hey, y'all, welcome to Geeking Out, the podcast about all geeky aspects of software delivery. DevOps, Observability, reliability, and everything in between. I'm your host Adriana Villela. Coming to you from Toronto, Canada, and geeking out with me today is Jennifer Riggins. Welcome, Jennifer.JENNIFER: Hi, thank you so much for having me on.ADRIANA: I'm super excited to have you join me. And where are you calling from today?JENNIFER: London.ADRIANA: Awesome. What I'll do is we'll start with some lightning round questions and then I'll get you to talk a little bit about yourself and then we'll go from there. Sound good?JENNIFER: Great, yeah, sure.ADRIANA: All right, let's do this. Okay, first question. Are you a lefty or a righty?JENNIFER: Righty.ADRIANA: All right, do you prefer iPhone or Android?JENNIFER: iPhone. Just because it's what I have and it's seamless. It's not a moral choice, but it's a convenience choice.ADRIANA: That's fair. Next question. Do you prefer Mac, Linux or Windows?JENNIFER: Mac. Same. Convenience.ADRIANA: Convenience is always very important. Okay, next question. As a tech journalist, do you lean towards Dev or Ops?JENNIFER: Oh, Dev. Well, no, that's hard. No, I would say either side. Yeah, because Platform Engineering is all about bridging that gap, isn't it?ADRIANA: Yeah, that's very true. Exactly. Okay, next question. Do you prefer to consume content through video or text?JENNIFER: Text for sure. Or audio more than anything. Podcast.ADRIANA: Yeah, I love me a good podcast. I have like way too many in my queue that I have to get through. Okay, final question. What is your superpower?JENNIFER: Connecting people, introducing different people that can help people figure out their next step or their next job or people should just know. People. Yeah.ADRIANA: Awesome. I love that. I think it's so important. I think people really underestimate the power of connection. All right, so we are on to the main event, the meaty bits, if you will. So why don't you share with our audience what you do with TheNewStack?JENNIFER: Okay. I have been a working writer since uni. I am not a trained journalist. I went for political science and I've been in the tech niche for 12 or 13 years. That includes both the marketing side and journalism side. I'm just a naturally good writer and good at explaining complex topics so that everyone understands, which is good because I'm geek by association, I am nerdy by nature, but I am not technical. So it helps me then help other people understand because everyone should be involved in understanding the future and how it's being built, especially as it gets more pervasive in our bodies. In our homes and our cars and then AI thinking for our behalf, on our behalf, et cetera.JENNIFER: And I have been writing for various as a freelancer, but with The New Stack for over eight years now, so pretty much their first year. And also I write for LeadDev and other blogs and then have software customers, things like that, helping them do their case studies or explain. I am not interested in funding, not interested in who's appointed CEO, not interested in crypto, not interested in technology precisely. I'm much more interested in the cultural impact of technology and what it's done. So I won't typically write about a new feature unless something extraordinary is about it. But I will write about once that feature is used and how it impacts people's lives, or more feature-driven like thought leadership, things like that.ADRIANA: Cool. That's awesome. So you mentioned that in university you did not come from a journalism background. So how did you find yourself writing for a living? Like you said, it came naturally. What gave you the first opportunity?JENNIFER: I've always been a natural writer, but I'm good at writing in that side. "Soy de letras," as you would say in Spanish. Math is how you would say it in English. And I was actually editor of my school newspaper and all, at university, so I was always involved in some way in writing and in helping other people write better things like that. So it's just a natural thing for me. I've always been able to fall back on writing.ADRIANA: And then how did you find yourself, like writing about technology then?JENNIFER: What else is there to write about? I think role was through Elance, or whatever it's called now. One of those Upwork, one of those freelance websites, and from there it spiraled. Something I'm good at explaining complicated concepts.ADRIANA: I think there's not enough emphasis on really being able to distill things in a very approachable manner, right? Especially a lot of docs out there, technical docs are so.JENNIFER: Complicated and incomplete at the same time.I think it's the most important thing. Critical thinking and being able to talk across that chasm or chasm between technology and business will be the greatest skill set and is so important, especially in this time of AI, because you need to be able to distinguish the bullshit that the AI we know is giving what, 52% of code generated by Chat GPT is wrong, but Chat GPT is very convincing because it was trained by tech bros, which have great sense of confidence and to sell bullshit. So it doesn't have to tell you when it's wrong. So in this time when we're entering AI and all this productivity mentality and everything, we need to be able to understand, be suspicious of what is working or not. And we also need to understand the business impact. So either side of it, whether it's business needing to understand that wildly expensive cost center of engineering and cloud, or engineering being able to explain and feel connected to that business impact and to understand, so everyone's going to have to explain to themselves. And Kelsey Hightower said at Civo Navigate, an event...he said, we have this weird, maybe it's a corporate throwback, where in tech we're like, I have this great idea, but I'm not done my slides yet, I'm not done my PowerPoint presentation yet. We'll wait to talk about it.But that's not how things work. People are storytellers. People need to be able to have conversations, even if it's expressing yourself in writing. I don't think it's necessarily very inclusive at all that everyone has to speak on stage or speak, but one-on-one conversations is still going to be a very important thing. And being able to write, even in Slack and be concise, so that's not my strong suit because I write very long features and things like that. But being able to express yourself in a way that everyone understands, because especially with AI, as we get into this interstitial age of prompt engineering, the next maybe two years, it's going to be the subject matter experts that are really important. So you won't need necessarily for everything, a coder. But if it's like building management or security in a building, maybe you need someone that actually has experience in that, who can work and partner with the developer to build something that's actually useful in AI.ADRIANA: Yeah.JENNIFER: So they need to talk to each other. And the people that may be deciding, especially with a chat bot, customer support and all, may have zero coding capabilities. So you need to be able to talk and communicate with them. And that's where the benefit from AI will come about. And it's honestly where we're going right now.ADRIANA: Yeah, I think the interesting thing is AI, in a way, keeps us on our toes because you almost have to be smarter than the AI to be able to pick out the bullshit, right? Because the minute you start trusting the AI and what it produces, that's what gets you in trouble, right?JENNIFER: Absolutely. And it's just different. We forget Chat GPT specifically is a large research project. It's not a tool. You are part of a research project. The tool is when you pay for like a private version of any of the AI tools that are trained on your context, your documentation, your processes. That's where the value comes. So if it's free, you should probably distrust it.JENNIFER: And also think about how bad that is for the earth.ADRIANA: Yeah, absolutely. I totally agree. Now, on the same vein of Chat GPT, I've heard initiatives from various companies where they want to replace a chunk of their written content with AI-generated content. What are your thoughts around that?JENNIFER: Okay, so in the world of documentation and things, I think it's very interesting. I think that is...documentation writers are super important, but there's also a lot of companies relying on developers to create docs. And in the 12, 13 years I've been in the industry, I started out a lot in the API space. Number one complaint was that there was not enough documentation. Yes, the number one thing developers don't want to do is write documentation. So having documentation embedded next to the code and somewhat AI-generated I think is very valuable. Human-generated media, things like that. There was a rumor 95% of media will be generated by AI by 2025 and all.I think we're having a real backlash about that. I know AI can't do what I can do, and I don't use it that much. I don't really use it. But my understanding, when other people use it and all, it's for the low value content. Have a proper conversation with someone to distill from someone that maybe isn't as easily expressing themselves because maybe they've got a very technical mindset. It can't have that conversation and draw out of them the true value of their product and then translate it?ADRIANA: Yeah.JENNIFER: Could it be useful if someone wrote an article themselves and then wanted to from that article spew out a bunch of social posts or something? It could probably be very interesting for that. Just very suspicious and controlling. You have to be anyway. But when you go through all of that, I don't feel my job is going to be in trouble. The people whose jobs are going to be in trouble are people whose lives live in Excel. Things that can and should be automated. The point is that we work on real problems. Boring, low level-coding problems will be automated, like repetitions.Creative work should get more creative, more problem solving. But then the boring stuff, I don't know what I could automate. I'd love to automate. Like invoicing, because I tend to procrastinate that because again, soy des letras. I'm not good at math, but then I don't trust the systems to throw that private information in there.ADRIANA: Yeah.JENNIFER: Also, we cannot forget that there's this unbelievable inequality that's being caused by data centers. It is causing a huge environmental impact. In west London alone, affordable housing cannot be built. There can be no new affordable housing in one of the largest cities in the world, one of the alpha cities, because too much power is being taken by data centers.ADRIANA: Wow.JENNIFER: To cool them down, et cetera. They're super polluting. Like, it's really bad. Note that I said affordable housing. So rich people who are leaving these plots empty and funneling money, because London's like a huge money laundering area, those are still being built and left empty. But people that truly need homes cannot get homes in west London because, specifically data center power. So I think we need to think about how we're impacting the environment. There's very interesting things going on for FinOps and optimizing your Kubernetes clusters, not getting in this habit of being double the amount of cloud just in case, but having things.And this is where AI is very interesting too, because AI can be a solution to help. It's always better to have the tool manage it than a human manage that, because if a human is responsible, they're always going to give more, just in case. They'll never give less, but they'll always more. So that's where AI can be a solution or part of the solution. But we should be putting far more pressure on anything we're paying for. We should be putting pressure as a customer that they are putting on data centers that are sustainable.ADRIANA: Yeah, I think we have to sort of move away from this mentality, as you alluded to earlier, of just more and more and more throw more at it, because it's like infinite resources. First of all, it costs money. If that doesn't deter you, which it should, then think about the environmental impact, which is just absolutely mind blowing.JENNIFER: And then that leads to another impact that disproportionately negatively affects people from underrepresented groups. Whether it's pollution in Virginia, which has a very underprivileged community, very impoverished community in Virginia that are directly...have hearing problems, have asthma problems, these are all problems. So yeah, I think we need to consider, in everything we do as tech storytellers, we need to consider the implication beyond the stereotypical developer, but we need to help them think about who will most likely be harmed by this and who will be more likely to be excluded or what being near.ADRIANA: Yeah, I completely agree. When you're writing an article, what inspires you? How do you decide what to write about?JENNIFER: It's 50/50 now because I've been writing so much about developer productivity and Platform Engineering, and, before DEI, but no one cares in 2023 about DEI. See the numbers. Sadly, diversity, equity, and inclusion is not a priority, so you have to do it surreptitiously, like by who you interview and stuff. Can't just write directly about it. I get reached out to a lot. I also see people's talks or use LinkedIn a lot. So there's all that.ADRIANA: And then the other thing I want to ask. You said that you do a lot of writing on Platform Engineering. What got you interested in Platform Engineering in the first place?JENNIFER: Oh, it's really a simplistic thing. I've been writing about and working in the Agile and DevOps space for a really long time. I write about culture side of tech, and like I said, in 2023, I see it in the data, I see it in traffic and all. Tech isn't even trying to pretend they care about diversity, equity and inclusion anymore. But you know what? Look at it while women, and that's probably the most privileged, minority or minoritized group in tech. While women make up about between 22 and 24% of the industry, there were 69% of layoffs. Black startups are not getting funding. I mean, it went from abysmal to 0.0002 abysmal percentage.ADRIANA: Wow.JENNIFER: People like Elon Musk and DHH from BaseCamp, they've made it cool publicly to not give a fuck about diversity, equity, inclusion. That means before it was informative...sorry...that means, before it was performative, but now they're not even trying to be performative. So there's that. And there's been a ton of cuts and layoffs. I see those cuts because there's two things. There's the last hired, first fired. So if they only started caring about diversity in the last two years, well, those people are going to be first cut. They also tend to be in roles like DEI, which were cut across the board.Accessibility cut across the board. Marketing, at least perennially, is cut when there's cutbacks, but tend to be more people from minoritized groups. But on the other hand, what's 2023 been about? A lot about tech layoffs, which means a lot of trying to do more with less. And then on top of that, the code is just getting more and more complex. The cognitive load is more and more extreme. And I think while we...we, not me.But the tech industry in general, doesn't seem to care about diversity, equity, inclusion, accessibility as much anymore, sadly, it does still understand, and I don't know that we can go back to, they've tried to return to office so many times and guess what? People are not happy, they're not productive, they're going to leave. Yes, the hand is more of an employer's market, but is still an employee's market across the board. And there's all these things where companies are realizing what statistics and data and journalism has said for years, that happy workers are more productive. And that doesn't mean massages and ping pong tables or foosball tables. That means actually finding purpose in your work, having visibility, not having even logically, from a nutty corporate standpoint, not having so many distractions and all the meetings blew up. So there's all of that. So there's this push for developer productivity because budgets are tighter, people need to make more money, staff is still bigger than it was a year, maybe two years ago. There was this irresponsible, cannibalistic growth for a while there, and it's kind of a correction, but the code has grown in the meantime too.The cloud native landscape is obscenely complex. So there's this idea we need to work on developer productivity, which is where Platform Engineering comes in. Instead of being a platform that we've had for... since codes exist. Like Cisco was making platforms back in the '70s. It was, you do this, you control this, which for some security stuff is not a bad idea for role-based access control and all that should not be optional. But the majority of the idea of Platform Engineering is that your customers are your developers and you are building a platform as a product where you are getting feedback from them constantly and you're building just what they need to get better. And then also it comes back to that whole docs problem. What is a huge problem? Who is breaking that developer flow, that getting in the zone is not being able to find things, googling it, going to Stack Overflow, asking a question on Reddit. Instead you've got this...we haven't even mentioned Copilot yet, but I think that for the developer audience has the most potential, because it's in with where 85% of repos are...in GitHub. So it's about them not context-switching as much and meetings actually having value, not having Agile.And then Covid just led to this multiplication of meetings for meeting. So Abby Bangser from Syntasso has my favorite definition of what Platform Engineering is, which it's almost like a physical platform you're supporting people on that takes care of the not differential but not unimportant work. So with DevOps, we went through this idea that you build, you test, you maintain, you do all of that, all the way to the cloud, all the way to release and all. But cloud is not differential to the average programmer, specifically to their audience, which would tend to be external users or customers. Security, very important, not differential testing. Very important, not differential repetitive work. Now it just should just be automated. So it doesn't matter anyway.And it's about...Spotify calls it Golden Pathway. I like calling it the Yellow Brick Road because if your developers wander off, they may go in a poppy field and go down a Reddit rabbit hole. But if Dorothy and them had stayed on the Yellow Brick Road, they would have been a lot faster. If Gandalf had given the eagles from the start, the book would have been a lot shorter. So why don't we do that? Guess what? If you had asked what Frodo would like? Oh, that's a new nerdy euphemism I'm coming up with right now, metaphor. But I think it works. Would have been a lot shorter movie, a lot shorter movie series, book series, and probably a lot more people wouldn't have died.So just ask your developers what is frustrating them and then start there.ADRIANA: Yeah, exactly. And there are so many things that frustrate developers.JENNIFER: And [inaudible] and searchability are always at the top of that list. They want to know who does what in a company, which again, comes down to collaboration and knowing people across the business. It's a positive thing to learn.ADRIANA: Yeah, absolutely. And there's another one. I think it came about from a question that you asked on one of the socials, which was something around, what are some of the developer frustrations? And I was thinking back to so many jobs where I started off...and onboarding and setting up a new environment on your machine is like the most fucking irritating experience ever. It's like, why do we have to keep doing the same thing over and over and over again? Why don't we have a streamlined process for setting up our dev environments when we start a new job?JENNIFER: Why would. Yeah, why would you even need to, why is setting up an environment useful for you to be doing? It's not helping the customer, it's not driving value. So Spotify, being like one of know, they created Backstage and outsourced it because they thought it was that important to standardize it in the community, which I like. But by them using Backstage, they got their developer onboarding time, which I believe they count as ten pull requests. Like that is when you consider productive. They went from 110 days to 20 days, pull requests because you just get people up and running. You give them what they need. You wouldn't give them a laptop and have them install Windows or install Linux or install whatever you want on your laptop. Give them the tool.ADRIANA: Yeah.JENNIFER: So just do that for all of the cloud because, and then you still give them the option. There will still be your 5% that want to engineer their way around a problem. And that's why you build it with APIs and you let people do their own thing. But maybe you don't need to support their work. They're at their own risk. They're on that poppy field, they're doing their own thing. But you'll support that 95% and that's okay.ADRIANA: Yeah. I really love your analogy of the Yellow Brick Road, because it really is all about like, these are your guardrails. It's there to protect you from yourself. Because we like to deviate. Sometimes we're not necessarily aware that that's not a great thing to do.JENNIFER: And you can still deviate. That's why you, as a Platform Engineer have to make something they want to use. And again, it comes all the way back to that tech storytelling, those early wins, the examples. Just the proof of good work is you need to make something they want to use. And then you have your customers who happen to be internal, probably more annoying, but you have a much tighter feedback loop. So you're going to get more direct feedback all the time. It's a good thing. It can just be probably a bit awkward for some people.Also, there's the problem that Platform Engineers are engineers, so they think they know best, which is not the point. And you just build something that they want to use, make it easy for them to stay on the path. So even the guardrails, I picture that car cannot really go past those guardrails. Follow the lines.ADRIANA: Yeah, it's like this is the path with some flexibility in mind, but you only have...JENNIFER: Fall off the cliff, and that is all you.ADRIANA: I think that's a perfect analogy. I love that. And the final thing that I wanted to touch upon, and you brought it up a few times, and I think it's actually a very important subject, which is DEI, which, as you pointed out, is the conversation around it has changed a lot, but the problem still remains. And it's kind of interesting because...I've had a number of conversations with people over the years, and after you pointed it out, I'm like, yeah, I guess it's kind of unfashionable to like, oh, let's have the panels of underrepresented groups talking about being underrepresented. Then it's like, well, as you said, we have to do it in a sneaky manner. But I think we do have to call it out for what it is because you go to tech conferences and I was a speaker at Observability Day, the co-located event for KubeCon North America, and there were three of us female speakers for all of Observability Day. And I was like, what the hell?JENNIFER: Could probably guess two of them just by knowing the handful of females or women that have access to that space and who are doing amazing work. But yeah, we don't need VIP bathrooms at tech events, we need representation. It's the only time we would be very happy to queue at bathrooms. Please, tech events.But like anything in the. When we're talking about open source, 3% out of what, 20 speakers or something for co-located day, it's actually not a bad percentage for open source because open source around 4% women and non-binary because it's toxic, because it's based on free work, which we do the brunt of anyway.ADRIANA: So true.JENNIFER: Women and people of color are far more likely to be doing free voluntary work and they don't have time for it. But then you lose the benefits of public code samples, of working with companies that actually are really big companies, like a Google or a Spotify or Atlassian, all these companies that support a lot of open source or access Amazon Web Services. These are companies that provide a lot of open source. But then if you can't go to these events, you can't work on these projects because you can't do free work. Open source is a huge problem. So it's always going to be worse. Which open source should I believe that open source should be free code, but I don't think believe in free labor, and I think that's a huge problem.ADRIANA: Yeah, absolutely.JENNIFER: You are a company benefiting from an open source project. You should be investing.Either find a way to sponsor that project or hire a staffer that contributes to that project as their deal, as their job, and just also focused on both technical and nontechnical contributions. Because again, we're back to documentation, we're back to the other big barrier to entry in open source diversity is that everything's in English. So you need people translate. Another use case that in probably 18 months will be very valuable from AI.ADRIANA: Yes, we take it for granted that we're English speakers, so we're like, yeah, of course, no problem. But I do remember, I think it was someone at KubeCon who was saying that they felt so shy about contributing to stuff because English wasn't their native language and they know incredibly smart, but they just didn't feel confident contributing to open source. And it just. Oh, my God.JENNIFER: Even in other languages, you need to know English too, to be a translator because it's the de facto language to translate to. But for example, Kubernetes, which Divya Mohan runs with someone else. I forget their name, sorry, but has organized for years the documentation translation, and it's across like 18 languages, or will be soon. Zero are in Africa. Are African languages zero?ADRIANA: Oh, wow.JENNIFER: Only about 2%, maybe 3%, depending on what you see of open source contributors and users are from Africa, which is about 19% of the world population and likely the geographic area that would most benefit from free and open and secure software, because typically open source is also more secure, more eyeballs, more people involved, et cetera. So it would benefit everyone, like, at an exponential GDP level, but because it's just in English...ADRIANA: Yeah. And it occurs to me also that even our programming languages...the syntax is in English!JENNIFER: And doesn't seem like that's going to change. Yeah, no, that is where AI, I think, will be interesting.ADRIANA: Yeah, it'll be definitely very interesting to see where it goes. Now, as we wrap things up, do you have any final thoughts on where you see this industry, our tech industry, going in the next, say, year?JENNIFER: That's it. It's a year, year and a half tops, because we're in this transition period where AI is still nascent, but it will very quickly advance and it will be much more useful because it will be context-specific, and I hope it won't be companies like...Telephonica in Spain fired, like, a huge chunk of its customer support reps because it's like, we can just use a chat AI. It's not great. I'm an HSBC customer, and I'm always like, give me human, give me human.ADRIANA: Yes.JENNIFER: It's not working. The Moby whatever, the chat bot thing, they. It's. It's not for me. I know a lot of people would rather talk to a bot, definitely, than stay on hold, but it's just not there yet. So we need humans in the loop now more than ever who have that subject matter expertise. We're not there yet, but we then need real humans in the loop feeding back into the AI, whatever it is, explaining to it, because people are still really nascent. But that's also part of the problem.A lot of companies...this was in my Spanish class. If I started taking Spanish class for the first time, at the YMCA. And that was our topic, Chat GPT. And I'm like, no, I don't use it. Other people are like, "Yeah, I use it for this and this." But then the Spanish teacher who's quite...kind of identifies as a Luddite, he says he pays for Chat GPT because then he gets the license, then he gets the right to his own content that he could one day sell. And I was like, "I didn't think about that." I thought about it more because a lot of companies don't have generative AI policies yet, which is ridiculous.Look what happened to Samsung. We're recording this in early December, I think in September, a coder didn't think about it and checked like a whole code base live in the public, free Chat GPT feeding like a bunch of private information in. And now Samsung's like, no more, no more generative AI, we're done.ADRIANA: Yeah.JENNIFER: [inaudible] behind, instead of every company needs like law firms. People are using it for stuff at consultancies. But if you don't tell people, like, do not put public information in here, do not put IP in here, or just pay the $20 a month for Chat GPT. I think it's five a month for Copilot and it's just a much better experience anyway. So pay for your tools and advise people how to use them. So I think just super important because I just think it's clear that AI is just going to be a part of our lives.ADRIANA: It is, yeah. And we have to be more mindful of how we're integrating it in our lives.JENNIFER: Because what is it? Copilot went GA early June [2023]. It's early December now...maybe mid June. By the time of the Octoverse Report, which I think was early November, late October, 92% of developers in the US were using generative AI.ADRIANA: Damn.JENNIFER: We're testing out. Like you can't take this away. They are finding value from, yeah, you can't take this away anymore, but you really have to have a policy. And it's shocking how few do in California or GDPR in Europe. I'm shocked we haven't had a big problem. I'm shocked it hasn't been big yet.ADRIANA: Yeah, it's been sort of...as companies realize that it's important, they'll implement it into their policies, but there's like, no...JENNIFER: [inaudible] And putting really wild stuff. I have someone I know in the journalist space who is much more technologically advanced than I am and not a native English speaker. So they had put a very nascent new technology...had written like a really deep dive article, evaluating it, explaining tutorial. They had thrown it into public Chat GPT to clean it up. Then they delivered the client. Three weeks later, their exact article showed up on one of those clickbait sites.ADRIANA: Oh my God.JENNIFER: They can't contact an editor, because...they can't contact a human being, because it's a fake human being, because it's like a clickbait site. But that site had found that this new technology was trending and they trained that site in it. They trained Chat GPT in it. And then it just took out their article.ADRIANA: Damn.JENNIFER: Don't put stuff that's not published or public in a public AI, whether Bard, it's Bing, whether it's Chat GPT, you don't know what's going to happen. Pay for it. If you want to play around with it, maybe. But even playing for fun, it still has an environmental impact that no one seems to care about.ADRIANA: Yeah, I'm so glad that you're bringing that up, because the more we talk about it, I hope the more it gets into people's brains that we cannot take for granted the things that we use. I mean, even Google, right? The fact that you're googling stuff, I mean, there are servers running things somewhere.JENNIFER: Google tends towards green energy more than the largest one, AWS. Leslie Miley, who was speaking as himself, but does work at Microsoft, at QCon, gave this wonderful in his keynote, just a really impactful talk. And he analogized the growth in AI to the US and maybe one of the world's largest infrastructure projects, which was the interstate road system, which specifically created red lines, which specifically was like, strategically kept people of color from being able to use buses to enter New York City and work, which still to this day in San Francisco or that area, the Bay Area, where we have all this, I assume is the most inequitable place in the world, where kids are three times more likely to have asthma, severe asthma, by six years old because of where these roads were built. So this idea, and it's happening again with the access to electricity, the access to data, the pollution, the access to clean water, because that's what's used...water is being used to cool data centers and it's happening around the same lines and stuff. It has this ability to create this great inequity and without diverse people and thought on your teams, people aren't considering it. And we know, again, one of those statistics, just like happy developers are more productive ones, more diverse teams are more innovative and profitable, but we've got our masks over our eyes again and not thinking. And that's where we are.So sorry to end on a bummer of a note, but let's think of the...I'm always back to there's a wonderful, Agile practice called Consequence Scanning from Emily Webber and Sam Brown. And I just recommend just doing a consequence scanning sometimes. Thinking about it's just simple questions like if this scaled, who wouldn't be able to use it? What are the good intentions we weren't thinking about? And what are some negative intentions or consequences that could happen because of this tool? This is one of those things with open source that even more because if you're being truly open source, your code could be used, I don't know, making another Kiwi Farms or another hate site. Hate farm, that's the consequence of open source. You need to think early on, "Okay, what if someone used this for evil?"ADRIANA: Yeah.JENNIFER: Negative consequences or what are the environmental consequences?ADRIANA: Absolutely. And I think that's really great food for thought. And I hope folks who are listening to this really take this to heart. And next time they use a tool like Chat GPT, they think about the environmental impact or even when they're using resources on the cloud, think about these things because it's so important and we've only got the one planet and time is ticking.JENNIFER: And don't trust the news. Like, these jobs like mine as a tech storyteller are not going away. We need more people. We need more people explaining in different ways, in different languages and different jargon so everyone understands what is being built and why and what the consequences are. Because a lot of people are just using.ADRIANA: Yeah, absolutely. Well, thank you so much, Jennifer, for geeking out with me today. Y'all don't forget to subscribe and be sure to check the show notes for additional resources and to connect with us and our guests on social media. Until next time...JENNIFER: Peace out and geek out, y'all.ADRIANA: Geeking Out is hosted and produced by me, Adriana Vilella. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who incidentally designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:Charity is an ops engineer and accidental startup founder at honeycomb.io. Before this she worked at Parse, Facebook, and Linden Lab on infrastructure and developer tools, and always seemed to wind up running the databases. She is the co-author of O'Reilly's Database Reliability Engineering and Observability Engineering, and loves free speech, free software, and single malt scotch.Find our guest on:X (Twitter)LinkedInFind us on:All of our social channels are on bento.me/geekingoutAll of Adriana's social channels are on bento.me/adrianamvillelaShow Links:Honeycomb.ioThe Four Tendencies, by Gretchen RubinChoose Boring Culture, by Charity Majors (blog)Helicopter Management, by Charity Majors (blog)Choose Boring Technology, by Dan McKinley (blog)The Advantage, by Patrick LencioniQuestionable Advice: "My Boss Says We Don't Need Any Engineering Managers. Is He Right?" by Charity Majors (blog)Performance Improvement Plan (PIP)The Engineer/Manager Pendulum, by Charity Majors (blog)The Hierarchy is Bullshit, by Charity Majors (blog)OktaCharity's Calendly for career adviceParse, Inc.Honeycomb Pollinators SlackDevOps Research and Assessment (DORA)OpenTelemetry specification has gone GAAdditional Links:Observability Engineering (book)Database Engineering (book)Transcript:ADRIANA: Hey, y'all, welcome to Geeking Out. The podcast about all geeky aspects of software delivery. DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And geeking out with me today...I am so excited to have Charity Majors of Honeycomb on! Welcome, Charity.CHARITY: Yay! Thank you for having me, Adriana.ADRIANA: I'm so excited. And where are you calling in from today, Charity?CHARITY: San Francisco. I just got home. I was in Charlottesville, Virginia, with my little sister over Christmas, and so I am newly home again, looking forward to a very quiet week between Christmas and New Year's.ADRIANA: That is always the best week for chillaxing, right?CHARITY: Nothing going on. This is why at honeycomb, we just give everyone the week off. Obviously, some people have to be on call, but why pretend you're getting stuff done if you aren't?ADRIANA: I know, right? Yeah, I fully support that. I totally agree. I think more companies should embrace that.CHARITY: Yeah. I don't feel like anyone should have to be performing that they're excited to be at work or like, we don't make people have a set number of vacation days or anything, but...That's the worst. If you're like, well, it wouldn't really be working, but do I spend one of my precious vacation days? Yeah, fuck it.ADRIANA: Yeah, I agree. Honestly, I get so much anxiety over vacation days, like, having to meticulously plan them and, like, oh, where do I spend them? And maximize vacation with family and school holidays. And there's, like, so many school holidays, right?CHARITY: Seriously, there's no perfect system. Like, if you do the unlimited holiday thing, people are like, well, but then you're not treating it like real comp. And people have stress about, are they hitting the right number of days or not? And people won't take it. But then if you have specific number of vacation days, then it's where do I spend it? And everything. So I guess if there's one thing that being a CEO CTO of a company has taught me, it's that people are going to complain no matter what. All you can try and do is pick what is genuinely best for your people that will really help you get as much work as possible done without asking people to fake it and do a bunch of. So, we've gone the infinite vacation route, because, all things considered, I think you kind of want to have a mandatory minimum. Like, you have to take two weeks off, right?ADRIANA: Yeah.CHARITY: And above and beyond that, it's like, are you getting your work done here? It's a standard. The company standard is about three weeks a year, but nobody's looking over your shoulder and policing you.ADRIANA: Yeah. See, I appreciate those policies, especially at companies where they fully respect autonomy, because there's the companies where it's like, well, it's unlimited, but we really only expect you to take like three weeks or four weeks or whatever, and it's like, so it's not really unlimited. Right. And that's disingenuous and annoying and very stressful. I don't know. I bust my ass and I need the time to chill.CHARITY: Yeah. But I will say some people will start taking five weeks, six weeks. But then the question that you have to ask them is, you're taking too much time. It's like, well, are you really getting your job done? And what's the impact on the people around you? Really?ADRIANA: Yes.CHARITY: Because, yeah, it isn't actually fair if you take eight weeks off. Anyone would understand if you have a health issue or if someone in your family is. We've had those situations. But if you're working at a startup with some intensity, we have VC money that's burning in the bank. You kind of can't get your job done, really, if you're not there for two months out of the year.ADRIANA: Oh, yeah.CHARITY: I think always trying to steer it back to the impact. Right. Can you get your job done and are you letting down the people around you, or are you being a real functional member of a high performing team? Those are the terms to have this debate on not how many days you're here or not. The other thing, unlimited time, is that it removes the aspect of scorekeeping and time keeping and quibbling about hours, because some people don't really care, but some people get really concerned about, well, am I taking 2 hours off here and 3 hours there? If I take 4 hours of that a day or not? And those are brain cells that I would really rather you just devote to solving the problems that we're paying you to solve, not to bookkeeping around your own anxiety or your projected expectation of someone else's anxiety about the hours that you're spending on your job.ADRIANA: Yeah, absolutely. I have to admit, the timekeeping stuff is so stressful, and I've been lucky the last three years. I have not had to fill out any timesheets, which has been like, oh, my God, my first job out of college was, like, consulting. So all of your fucking hours are accounted for.CHARITY: Oh...ADRIANA: So everything and even your downtime, right? If you're in between projects, you got to charge it to internal thing. And it was like, yeah, I lasted four years.CHARITY: Oh, honey. I don't know how! One of our company values is we hire adults. And I actually think about that. It's as much about us as it is about the people we hire. It's like, are we treating people like adults? Do we expect them to manage their own time or not? And of course, the difficult points come. I think as an industry, we're just terrible at figuring out how to really take people on as apprentices and turn them into fully-fledged employees. I mean, there's that middle section that takes, even for a fresh college grad or someone entering...It takes five to seven years, I think, for you, really, to bring someone on and bring them up to a level of senior engineer and teach them all these things.But you can interpret it, our value as you're on your own. You better come fully baked because we're not going to help you, which is not what we're trying to project or do. But it's challenging, no?ADRIANA: Yeah. It's so challenging, like coming out of school, right? Trying to figure out where you fit in. And it's also kind of, for me, it was like a bit of a mind fuck because I was like the goody goody. Like, I will do all the assignments. And marks were everything. And then you go out into the real world and it's like, yeah, bye bye. That did not apply. For me, it was a massive adjustment and I kind of sucked fresh out of school, like my first couple of years in the work world trying to figure out, what do I do? What do I do? There's like, no marks. Not in the standard sense, right?CHARITY: No, of course not. You must be an upholder type. Do you get a lot of satisfaction out of checklists? Like your own checklists and the checklists that people do?ADRIANA: I do, I do. My own checklist. My whiteboard next to me. It's mostly clean now because of the holidays, but it had my to-dos...but I've had to learn to roll with it. I had to be a lot less uptight than I was in school, because I think you just have to, in the work world.CHARITY: Well, because you learn eventually that if you want to be successful, it's not actually about checklists, it's about figuring out what matters to you and what matters to other people and then figuring out how to creatively achieve those goals. And the checklists are there as a tool, right? I'm not telling you anything you don't know.ADRIANA: Yeah, exactly. Yeah. I completely agree. And I think that's a lesson that comes so much more easily for some than others for sure. Especially. I've hired a couple of interns in my past life and trying to steer them in the direction of, like, chill. Let's relax. Let's just focus on getting the work done and learning cool shit.CHARITY: In a lot of ways, though, I would argue that the upholder is the easiest type of person to onboard because they're motivated by everything.ADRIANA: True.CHARITY: So when I use the term upholder, I don't know if you've read the book, "The Four Tendencies"? It's this book that it's super cheesy and I don't want to get anybody's expectations up, but it was actually really pivotal for me and Christine [CEO of Honeycomb] and finding a way through our relationship because she's an upholder. I'm the opposite. I'm a rebel. Which means that I reject all of your checklists and my own too, called checklist. Basically, it's about motivation. And there's only four possible types.It's a two by two, right? It's like your own motivation, like what motivates you and the goals that you set for yourself and then the goals that other people have for you. And you can either be super motivated by both or you can be what's called a questioner type, which you can't really give a fuck about other people's expectations. But if you care about something, then you can hit that goal every time. And then there's the type that needs a gym buddy because you struggle to do the things that you set for yourself, but you respond really well to external structure. And then there's the type that rejects all of the structures. And that's my type. And this was really helpful to us in just like, sort of because Christine and me are just such polar opposites that she was just like, who the fuck are you? How does your brain work? Why is it that I give you this perfectly formed challenge and you're like, "Fuck all your challenges." And I'm just like, "Why are you telling me what the fuck to do? Don't you know that's the easiest way to demotivate me, is to tell me what to do?"And so it was really helpful because this book actually has these almost, like, examples of, if you're this type in a relationship with this type, here are some conflicts and conversations that you might have if you're in a working relationship and you're this type paired with this type. And it was just like, oh, my God. Some conversations that I had had with my partner, like almost word for word, some conflicts Christine and I had had, almost word for word. It was just like, here are some tools for getting around them. So I really like it.ADRIANA: That is so helpful. It's funny, because I think the way you describe yourself is how I would describe my daughter, too, to a certain extent, because when she was in preschool, her teacher could not teach her, and she realized that the way to teach her was not to teach her, but to teach her friends. And then it would cause Hannah to go over, oh, that looks interesting. So she's like, don't tell me what the fuck to do. I'm from Brazil. And I'm like, oh, it'd be so cool if you learned Portuguese. She's like, "No." What did she do? She learned German.CHARITY: That is how you deal with rebels. You have to rely on them to find their own intrinsic motivation, because if it becomes part of their identity and part of who they say that they are, then you can't stop them.ADRIANA: Yeah, exactly. Yeah. So I'm like, you know what? You do you. I embrace that. And I think she's happier for it. I'm happier for it.CHARITY: Everyone should be happier for it. As a manager, part of what you have to do is, I feel like, as a manager, in the beginning, we try to give our reports the experience that we wish we had had. For upholders and for...I can never remember...the obligers. Obligers are the ones that need the external structure. You're really giving them a gift. If you give them a structure or if you give them regular check ins and you let them know what the expectations are, you're giving them a huge gift, and they will rise to the occasion and they'll thank you for it. And if you do that for rebels or questioners, you're insulting them.That sort of versatility. And it's not just managers, of course. It's anyone who's, like, in a senior plus position, where what you need to do depends a lot in influencing others. Just sort of having a mental map of how other people respond to sort of motivations is super helpful.ADRIANA: Yeah. I actually remember reading one of your blog posts on, like, I think you're talking, like, being manager and trying to make everybody happy, but it's not also about being their buddy and making everybody happy, but also, you do have company goals to fulfill. And so to what extent do you protect your team, but then don't end up doing the things that need to be done, which I think is such a common pitfall for new managers, because for me, certainly when I first got into a management role. I'm like, this happened to me.ADRIANA: I'm not going to let that happen to my direct reports. I am going to be the best manager that I can possibly be. Right. It can kind of blow up in your face if you're not careful. Like, I wanted to be friends with my direct reports. That did not work out in the long run. Initially, it was like, yaaaay. But afterwards, it was like, no.CHARITY: We're always overcompensating for our own experience.ADRIANA: Yeah, exactly. And in the end, I think we learn, right?CHARITY: Yeah, exactly. Eventually, hopefully, we find a happy medium. I think about that so often when thinking about diversity issues in the industry or about management or that it's natural for there to be like, this is a young industry. This is a very young profession. For as old as some of us feel like we are, we're still like, there's been... When I was coming up, we didn't talk about women in tech. There was a few of us that were just, like, quietly there, wearing men's clothes and just sort of pretending we were straight white dudes. And so there was a lash, right? And then there was a backlash.And it swings. I'm not going to say too much about how sensitive I think some people are, but I understand why they are. I understand why they are. And also, that's not where we have to end. That can't be where we end up. We have to end up in a place that is less reactionary on all sides.ADRIANA: Absolutely.CHARITY: The goal of our businesses and our companies, this is something I've been thinking about a lot. The few times that I feel like the honeycomb culture has gone off the rails a little bit, is when we've kind of lost sight of the fact that we are here to serve our customers. We are not here to have the most diverse company in the world. We're not here to give people the best work life balance. We aren't even here to give everyone the best employment experience of their lives, which early in our, when it seems for so many years like we were going to fail, Christine and I would console each other. We'd be, you know, if we go under tomorrow, as we think we probably will, at least I think we've done a good job of giving a lot of people an experience that will set the know so they won't accept shitty jobs for the rest of their life. But now that we're hoping to be around for a long time, we can't forget that we are here to serve our customers. The decisions that we happen to think that a lot of these things go in harmony.Treating people really well means we treat our customers well. Having people who are happy at work. We believe in having healthy businesses, which is a lot of people's complaints. They see symptoms, but what they'reacting to is the fact that the business is not healthy. The way people are relating to each other is not healthy. I wrote this other blog post a while ago, I don't know if you saw it about, "Choose Boring Culture"?ADRIANA: That sounds vaguely familiar.CHARITY: You know, because Dan McKinley wrote that blog post that was hugely influential on me about choose boring technology where he's like, you know, as a startup you get three innovation tokens. Choose wisely. And I feel know the same is true for culture and businesses. And like, we stand on the shoulders of...you know, a lot of people, a lot of really smart people have figured out things about how to make companies work well. There's this great book by Pat Lancioni called the Advantage, which I think of as like the James Madison of business and organizational structure. He's incredibly innovative thinker and he makes things very simple. But he's like, the advantage increasingly in corporations is not your widgets. Because everybody's widgets are getting so good. It's how healthy is your organization, which means how much of your people's creativity are you really taking advantage of? How much of their creativity do you feel free to bring to work? Is your organization equipped to absorb it and to change from it and to react to it? Are you able to keep people who are passionate about their work? Do you let people go who are detracting from the culture? And he's like, it is amazing how poorly most organizations are run to this day.So choose boring culture. I think in a lot of ways, companies don't have to make their companies interesting and fun because people will do that. People have so much fun, creative energy in themselves. You just have to create a boring place for them to work where they can do their best work and they'll come up with all the fun stuff.ADRIANA: Oh, I love that. That's so cool. You touched upon something that I am a huge proponent of, which is like, letting go of people who are not adding to your corporate culture. Because I think there's this tendency, I think, in our industry to hire rock stars and kind of ignore the shittiness and their personality because, oh my God, they're the best of the best at blah. Right? And I've personally experienced a couple of incidents in my life where if you have somebody who is constantly just being negative on your team, no matter how good the rest of your team is if they're like, poo pooing everything, it sullies the culture. It's like a poison pill. And it's not like, oh, I'm going to fire your ass. It's like, well, perhaps this team might not be the best for what you want to achieve. Perhaps I can help you find a position in another team in the company. Because it's just poison.CHARITY: I think it starts with not having kid gloves on. I don't think you jump straight to firing. I don't even think you jump straight to moving. A lot of these people have never really been told no in their lives. And some of them can take it, some of them can. But I think you owe it to them to figure it out, right? To start giving feedback consistently and regularly working with the person. And this is something that I think can be really frustrating to people who are. When it looks like management is doing nothing right, because it looks like, I know that people at Honeycomb have felt this way at times, because it looks like they're just kind of being shitty and they get better and then they don't.And it's always a judgment call. And I would actually agree that we always probably wait a little too long in general, but we waited a little too long with everyone. And I would take that over being a little too fast to fire people, because I think that that even more trust. But, yeah, I agree. If they can't bend, if they can't change, if they can't understand that the smallest unit of software ownership is the team, it's not the person. It doesn't matter how great one person is, because one person can't own software. It's all about, are you contributing to the overall greatness of this team? You can bend your rockstar talents to that, but if you're not willing to, or if you can't, then there's no place here for you. I'm sure you can get paid a lot more money somewhere else.ADRIANA: Yeah, absolutely true. Absolutely.CHARITY: Sorry, go ahead. I didn't mean to cut you off.ADRIANA: Oh, no, I was just saying I agree with you, but I think that.CHARITY: Letting go of people is hard, and I think that it comes in all forms. I think that it's really discouraging to people who are on a high performance, who want to be on a high performing team, when someone isn't really showing up and who consistently isn't showing. The person who's like, consistently taking six weeks of vacation when everyone else is taking three or four, or the person who is kind of half asking it. And all of us half ass it sometimes, right? But people can tell you work on a team for a while, you get a real good sense of how hard everyone is working, how much they're trying. Sometimes it comes in form of, this is almost some of the most heartbreaking ones of when you've got someone who's very junior who just isn't working hard enough. And it's like we kind of don't have the language to tell them that. Because on this pendulum, we're so far over to the side of, you shouldn't be like, work crush code. It's almost like we've kind of lost the ability to tell people, no, really, you're probably not going to make it if you don't put in a few more hours and if you don't have a little bit more grit.And some people don't want to work that hard, and that's fine, but you aren't automatically granted a job based on however hard you do or don't want to work.ADRIANA: Yeah. And it's such a tough conversation to have. I had someone in a previous team that I hired on as a senior person, and then she was, like, scamming on my. She was scamming on everyone else. She would just pretend that she was doing work by, like, oh, let me attend meetings with so and so. And meanwhile, I'd hired this junior person who was working like she was working at the senior level. And it was so frustrating. I was trying to have the conversations with the senior person saying, listen, I want to help you. How can we work together? But she got offended. And these conversations are so hard to have because we all perceive differently how we're doing. And in her mind, she was doing just fine. How have you had those conversations in the past with people?CHARITY: Oh, it's really hard. There's no version of this that isn't hard if you care about people.ADRIANA: Yeah.CHARITY: My most recent blog post was about why anyone should go into engineering management. Because it's a hard fucking job. And the answer is, because we need them. Because we need them desperately. Like a team with a great engineering manager builds circles around teams without one. And the other reason in my piece, I said is that it changes you as a person, and it gives you these skills that a lot of us didn't learn when we were growing up about how to be honest and how to have hard conversations and all these things. But as to your question, how do you go into this? The number one thing I think is no review should be a surprise. You should be having this conversation consistently, which is a hard thing to do because it makes people feel demotivated and frustrated.But sometimes they have to feel that way. We've instituted a rule at honeycomb that if you're thinking of putting someone on a PIP, if you're thinking of, you have to literally say the words, your job is at risk because it's so tempting when you're face to face with someone who you really want to succeed, to soft pedal it or for them to feel upset and for you to kind of walk it back, or for you just to use words that let them walk away thinking something that is not what you want. And there are tools you can use to make sure. You can write up an email afterwards to be like, just to be clear, this is what I saw. This is what I'm saying. This is what you're hearing. But I really do think that one of the most important tools we have is just being explicit because they can file it away. We all have such infinite creativity when it comes to explaining away things that we don't want to hear.And we can be like, oh, my manager is kind of a bitch. Oh, they're just in a bad mood. Oh, they're just kind of riding me lately. Oh, it's because of this thing. But this will be over. And I feel like if something really isn't trending, well, we have a responsibility to be more of a dick. We have to be the ones who kind of put our bodies in the breach and be like...and just sit there and deal with their reactions, which are going...They're going to have negative feelings. And it's really hard to sit with someone else's negative feelings who you are the proximate cause of. It's really hard, but you have to do it. It is the best thing for them to do it, to let them know this isn't just a small thing. This isn't just a flash in the pan. You are not succeeding. You are not on a path to succeeding here. You are on a path to, your job is at risk. Honestly, that's the kindest thing you can do for someone.ADRIANA: Yeah, that makes so much sense. And you're right. It's so hard to get those words out. Like, "Your job is at risk." Yeah. And I've worked in organizations, too, where pussyfooting around the topic was like kind of the cultural norm, and so things wouldn't get said that should have been said, and you don't have the favorable outcomes in the end.CHARITY: Yeah. And then people feel stabbed in the back, understandably. I would, too. They go...walk away going, "If they had just told me, if I had only known." And that is the worst outcome. That is the thing that I always remind myself of when I'm just like, I love this person. I don't want to be mean to them, but I cannot take it if they walk away feeling like I didn't tell them, like I stabbed them in the back by not making it perfectly clear that they're not performing and their job is at risk.ADRIANA: Yeah, it's definitely something that I wish that I had done more of in the past, and I try to remind myself of it, but, yeah, I think that is absolutely the right thing.CHARITY: And to your point earlier about being people's friends, you can absolutely be friends with your direct report, but there's a line there. There's a boundary there, and there's a point at which you're not their friend. It's just like being someone's parent, right? When things are going great, yeah, you act like friends, but they have to know that when it's time for you to be parent, you're going to be parent.ADRIANA: Yeah, exactly. Because otherwise they will take advantage of you.CHARITY: Right. They will completely take advantage of you. It's human nature.ADRIANA: Exactly. And you will let your guard down, too, right? Because they're like, oh, "I don't want to hurt so and so's feelings, otherwise they won't love me." And it's like, you kind of have to get over that as a manager. And it's hard.CHARITY: It's really hard. It's really hard. And it's always a matter of judgment. It's always a judgment call. And you have to know that after you've had that hard conversation, chances are they're going to go tell other teammates a version of it that makes you look bad and them look great. And you can't do fuck all about it. You have to sit there and take it and hope that the relationships and the trust that you have built up are enough that people aren't going to just automatically believe that other person. That is the hardest thing about being a manager to me.CHARITY: That right there, knowing...is when I know I can't say anything.ADRIANA: Yeah. And risking, as you said, having people say, well, management doesn't know what they're doing. Oh, my God. Because as an IC in the past, I was like, management clearly doesn't know what they're doing, and then...CHARITY: Clearly doesn't know shit.ADRIANA: The first time it happened to me, oh, my God, I want to go cry. Like I'm trying everything to make you happy.CHARITY: Yeah. This is why I feel like my dream vision for the future of engineering management is that more people do it. But people don't do it. They don't do it as a career. They do it as a tour of duty, because I feel like having ex managers on the team, it's like a game-changer, because whenever the dynamic is ICS versus managers, which always happens. Comes and goes, but it always happens. It's so helpful to have an ex-manager there on the IC side who could go, okay, kids, it might be this. It might be this. It might be this. Do we trust this manager in general? Okay, well, let's not jump to the automatic conclusion that they're just an idiot or they're just, like, being manipulated by the upper or whatever. They're the only voice in the room who can talk people down off a cliff and remind them whether to have some trust. And it's such a game changer. It is so wonderful.ADRIANA: Yeah, that is so true. And it makes so much sense. I even find myself in positions after I've been a manager, and then being now an IC...whenever I get comments...CHARITY: It's nice!ADRIANA: Yeah, it is nice! And sometimes I have my manager apologize, "Oh, I'm so sorry. Blah, blah, blah." I'm like, "Dude, I totally get it." "It's fine. No worries."CHARITY: You're able to give so much better support and understanding to your manager than you ever could have without that experience.ADRIANA: Exactly.CHARITY: It's so grounding and validating for them to have someone who sees them.ADRIANA: Yeah. And especially, also when you have that nice rapport with your manager where you have that ultimate trust, where, okay, it might seem like they're riding you hard, but then you're like, oh, my ex-manager brain has said, okay, "I have a good reason to trust them. Take a step back. Let's look at the big picture." And, yeah, it's cathartic and it's eye opening.CHARITY: Everyone wins.ADRIANA: Yeah, exactly. No, sorry. Go ahead. No, please.CHARITY: I often hear people who are first-time managers who are, like, anxious or like, if I go back to being an IC, will I ever get the chance again to be a manager? And I'm just like, "Oh, grasshopper, they can smell it on you. You will be fighting off manager opportunities for the rest of your career." Have you found this to be true? I expect you have.ADRIANA: Yeah, I have. And it was funny because after I read it in one of your blog posts, I was like, oh, yeah, so true.CHARITY: Yeah.ADRIANA: I mean, it's on your resume. Yeah.CHARITY: Just the way you come across. I've also said that the fastest way to mint like, a shiny new staff engineer is to take a senior engineer and put them in management for a couple of years. Because the way you present yourself at work, the way you approach problems, you have such a better sense of the business, even if it wasn't on your resume. This is why some people get to be managers early and often, because for whatever reason, they already have some of those skills. But once you've been a manager, it's written all over your face that you understand.ADRIANA: Yeah, very true. Now, here's a question for you. What's your take on folks who have gone into management at a really early point in their career, becoming a technical manager for a technical team when they don't have that many years of actual technical experience?CHARITY: I think they are not well-served by this. I often see this happen to women, especially, and I think it's often intended as a compliment and by people who genuinely are trying to do they want to help the industry. They know that there needs to be more women in leadership and management. And so they're like, here's this person who has social skills and also some engineering skills. So we'll just...I think everyone has the best of intentions, and I think it really does not serve them because it's often a one-way...it's a one way-ticket, right? Because you don't have the skills to be able to go back and pick up coding easily in a couple of years. I think you also don't really have the skills to be a great manager.Honestly, my recommendation to them would be get back to coding as quickly as you can or climb the ladder. If you choose to climb the ladder, then those skills are less relevant. But I wouldn't be in a rush. If you're 25 and you're a manager getting offered a director position, I would look at that cross-eyed. I would be like, because, yes, it is probably a compliment, but is it the right thing for you? I don't know. I mean, if you play out over the course of your career, you've got a 30, 40 year career. There's no rush. And the people who really excel in those senior leadership positions tend to be ones with deep roots, not just a very shallow.And there's so much to learn, right? This is not to say that there's not anyone out there who's climbed the ladder in a hurry and not regretted it, because there probably is. But the people that I know who have done it have, by and large, profoundly regretted it. You know, I wrote about my friend Molly, who's an engineer at Honeycomb now, and she was one of those people. She super bright, straight out of college, became an engineer, became a manager, became a director. Shot up. You know she was a VP, she was a director, she was an EP. And she came to Honeycomb to be our head of...VP of customer success or something like that. And she was so unhappy.And she would make all these wistful comments about how she wished she could be a software engineer. She wished she had done that. Eventually, her husband, he was an early member at Okta and Okta IPOed. And so suddenly she was like, "Wow, I can do anything I want with my life. I want to be a software engineer." And so she became a support engineer for us, and she just started writing code on the side. She started picking up some PRs. Now she's a software engineer on the team, and it's been hard.She's never been happier, though. And I'm proud that Honeycomb is the kind of place that can support someone in doing that, because I think the opportunities to do something like that are few and far between. There are not many places we'll take a flyer on someone who's middle-aged and wants to go back to software engineering. But if you think of your career as a long game, you don't want to amass a bunch of titles, especially titles that are kind of empty because you're not getting a...I would...I would venture to guess that you're not getting a really high quality offer to be a director or a VP at age 27. It's really mostly the title. You want to amass yourself a solid base of experiences and skills, and you want to have shit to draw on as you climb that ladder so that you can help people better.So the thing that I do want to guard against when I'm talking about this, I'm speaking to people who are early in their career, who are facing these questions. I don't want to make it sound like it's too late and you're screwed if you're already in this position. In fact, if you're in that position, if you'd like someone to talk it through, reach out to me. I have a Calendly link, calendly.com/charitym/advice, and I'm always happy to talk through interesting and tough career conversations with people. You have skills, you have assets. It might not be a super sexy path, but you can find places that will take advantage of the skills you have to offer while you kind of work your way up from the bottom again, if that's what you want to do. I'm sure you can do it, but it's easier if you do it right the first way and become a solidly senior engineer. Seven years really is the minimum, I think, before you become a manager.And if you really want to be able to manage other senior engineers, you need to at least be able to speak the language and be able to roll back on it.ADRIANA: Yeah, I fully agree with you on that. I was thinking back to my own career. My first job out of school was as a consultant at Accenture, and the career path was basically like, you must pay your dues as a developer, and you shall be rewarded with a management position. Right? Yeah. Right. So we're all kind of brainwashed to think, oh, my God, if I'm not a manager, by 27, 28, I have failed at life. Right? And I hit this crossroads in my life where I was being groomed to be a manager. I didn't have the manager title, but they threw me on some engagement where I was managing three teams at once. I was doing a shitty job, and I'm like, I was miserable, and I'm like, what do I want to do with my life? And so I decided...I left consulting. I took on a job as a software engineer. It was a lateral move, but I was so happy, and it was the best thing for me because my thought was, how can I manage these people if I don't know enough? I just didn't feel right for me, so I'm happy I did that.CHARITY: Good for you for listening to your gut. I think all too often we talk about impostor syndrome, and we try to talk people out of it. I often think if your gut is really eating at you, that something is wrong. You should listen to that. You shouldn't just go, oh, everybody, there's impostor syndrome, and then there's just, like, the feeling in your stomach that you're not really setting your future self up for success or that you aren't really equipped to do the kind of job that you want to be able to do in this role. And I think that is not something to be brushed aside lightly.ADRIANA: Yeah, I definitely agree. Listen to your gut, because it's telling you something. One thing that I wanted to ask was, when you were building Honeycomb from the ground up, did you have sort of lofty aspirations of how you wanted things to be?CHARITY: Ha!ADRIANA: How was...the initial thoughts versus how it turned out?CHARITY: I 100,000% expected us to fail like the plan was to fail. So I was never one of those kids who was like, I'm going to start a company. Because I always kind of low key hate those people. It's like, "Oh, you're too good to work for someone else." I'm not too good to work for someone else. I was a serial dropout. I'm the opposite of you, right? I didn't collect all the awards. I didn't check anything off.I dropped out and I dropped out again. I dropped out again. And so I never had a pedigree. Nobody was ever going to give me money. Then I was leaving Facebook, and the first time in my life, I kind of had a pedigree. And so I was like, well, I can't waste, like, on behalf of all women and queers and dropouts everywhere, I have to take it and run with it and do something. But I was super burned out. And I was like, well, I guess I have an idea, but I'll go heads down the corner, write code for a couple of years, and then we'll fail.And I'll open source it. Then I'll have my tool to use. Hee haw! That was really the grand vision. And I would say Honeycomb has been around for eight years as of January 1, but we had many near-death experiences. Now, we hit our $40 million ARR mark this month, which is exciting. We're hoping to get on a path towards an IPO. But for the first five years, I think we wobbled around between 5 people, 12 people, 30 people. We did layoffs down to 15 people again. We were a skeleton crew wandering in the wilderness. In retrospect, I realized that we were creating a category and we were writing the database and all this stuff, but it just felt brutal. It just felt like failure was around every corner. And most of those corners were right. We did fail most of those corners. There are several just, like, near-death experiences that we had, and we made it through.And now I, for the first time, am not thinking we're going to fail. But no, there was no grand vision. There was no grand vision at all. There was just, like putting 1 foot in front of the other and feeling like I was failing the people that I loved most almost every single day. It was brutal. I will say, though, that Christine and I, a little bit older than your average tech founders, especially me, and turns out we have very strong opinions. And we learned a lot of lessons at previous startups. We were at, like, at Parse, which I loved working at Parse.Parse is where I learned about the importance of design, about marketing. People loved that product and I loved working on it. Before Parse, I was like, I'm just a backend engineer. I don't care what the product's about. I'll work on anything. Parse is where I learned that, of course, that was never true. But Parse never really had a shot because the founders never really tried to build a business. They tried to build a great product, and they did. But then around series B, they had a marketing person and a couple salespeople.We weren't bringing any revenue. They had to sell. Their destiny got taken out of their hands because they had no other choice. And so Christine and I, from the very beginning were like, we want to build a business. We want to build a business. We want to build a product that people want to pay money for. We're not building freebies. We're going to try and monetize on the other end of the pipe.We are building a product. We're building a business. And I had a lot of just, like, very strong opinions about the kind of culture we wanted to build, just about how...in the beginning, when we were interviewing engineers, if anyone talked, not even dismissingly, about go to market functions like sales or marketing, even just sort of, like, almost alienated, just like, "Oh, well, that's them. We're us. We don't understand that." Those weren't our engineers, because we don't need to hire engineers who wanted to build a business with us and who weren't going to create that us versus them dynamic that makes all great business people in the valley feel like second-class citizens. So, yeah, I would say we discovered the grand vision along the way. It really wasn't there from the beginning.ADRIANA: And as a follow-up, you know, one of the things that I admire so much about Honeycomb is you build such a lovely community around your product. Your customers truly, truly love it. And we met because I was asking so many questions in the Honeycomb Pollinators Slack. At the time, I was exploring Honeycomb as a potential product that the company I was working for might switch over to. And everyone was just so genuinely nice in helping me understand this Observability thing that was so nebulous. How do you build that thoughtful community? Was it something that you sought out to do from the get-go? Is it something that organically grew?CHARITY: If you ask any founder, they'd say they're trying to build that, right? So I think the questions were like, "Why were we more successful than many others?" I think a lot of it has to do with just...and if you had asked me if I would be talking about values and shit, like a year ago or a few years ago, I'd be, like, rolling my eyes, because I've always hated when people are like, "Values," because most businesses are just like. I don't know. I get really cynical about it, but I feel like we are our customers, and our customers are us. We built this product to solve a real problem that we are having. And it is more important to us that these problems get solved than that Honeycomb is successful. I think I can say that about everyone there.We would love to be successful. We'd love to make lots of money and all this stuff. But we see the pain that so many teams are in, and we know that we have a way to fix a lot of that pain, because we've seen our customers do this over and over, and we hear what they say about how no one else could do this. And we had the advantage of designing and building this 25 years after metrics began dominating the landscape. So we build on the shoulders of giants, like I said earlier. So I feel like it's easy to be a true believer, because we're not just trying to sell something. We're really building something that really changes people's lives. And it's easy to get starry-eyed about that.It's easy to be a believer when you're all on the same page about fixing problems, not just about trying to tweak your messaging or your marketing or your sales or something. I think people, Honeycomb, are generally very passionate about solving the problem, and it's very exciting to see them. I mean, the product does what it says on the sticker, which is very exciting, because almost no products do. Most products are hyped. If anything, Honeycomb is underhyped. It does so much more than we've been able to explain to people, which is why our churn is like nothing. We win, like, 80% of our tech evals, which the industry standard is, like 30 or 35%. Once people see it on their data, you cannot pry it out of their cold, dead hands.One of our best sources of leads is when engineers change jobs and they bring us with them, because once they've tried developing with Honeycomb, they can't go back to not having honeycomb. And this is all stuff that it's hard to explain to people in words, but once they see it, it clicks. And so, really, our core challenge, over the next year, we've built the product. Our core challenge is figure out how to get more people to click with it faster, because we know that once they've seen it. The deal is done, but it's still a very hard problem.ADRIANA: Yeah. The other thing that I think is very interesting about Honeycomb is it's not only are you building a product that people are excited about, but you've also really turned the whole area of Observability on its head. I'd like to think that it was Honeycomb that sort of gave Observability...Observability became what it is because of what Honeycomb has done. I mean, you've spent a lot of time talking about Observability. I mean, honestly, that's how I got dialed into what Observability was in the first place, was catching your Tweets. Yeah, if you could say a little bit more about that.CHARITY: Yeah. Like, Christine and I are not marketing people. It turns out what we were doing was category creation. All I knew was that we were trying to build something based on an experience we had had that had changed us as engineers, and we knew that it wasn't monitoring. And I spent months just sort of, like, testing language, trying various things. And one point, it was July in 2016 that I Googled the term "Observability", and I read the control theory definition, and I was like, "Oh, shit. This is what we're trying to do. We're trying to build something to let engineers understand the inner workings of a system, no matter what's happening, just by observing its outputs."So, like, working backwards from that, what do you need? Like, you need the high cardinality, you need the high dimensionality and all this stuff. And I feel like that definition really took hold for about three years. In 2019, 2020, maybe 2021, all of the money started rushing into the space, and suddenly, anyone who was doing anything with telemetry was like, cool. We do Observability, too, which, on the one hand, is like, it's a good problem to have. It means that what we were talking about really resonates with people. And at the time, I was naïve enough to think that, oh, well, they're co-opting our marketing language, but surely they're building the same technology under the hood. It's just a matter of time until they release it. I don't believe that anymore.I think all they did was steal the marketing language, and I don't think they actually have any plans to. I think that, like, Datadog in particular, their business model is centered around having all these different SKUs, right? A different product for metrics, for logs, for tracing, for profiling, for security, and they've got too much money invested in. The problem is that the experience degrades for everyone if nothing connects all these data sources. People are paying to store their data again and again and again and again, but nothing connects it except the engineer who's sitting in the middle just trying to visualize or visually correlate. If that spike is the same as that one, it's fucking broken. My hope is that there will be new startups that are entering the space. So I've kind of given up like, okay, Observability now means, and this makes sense, I'm actually completely on board.Observability, instead of having a strict technical falsifiable definition, Observability is a property of systems, right? A system can be more less observable if you add some metrics, great, you're more observable. But what we're seeing in the field is that there's a real huge step function difference between, let's call it Observability 1.0, which is about metrics, three pillars, right? And Observability 2.0, which is based on this single source of truth. And it's not just the technology, because o11y 1.0 is very much about MTDR, MTTD reliability, uptime. It's a checkmark before you send your code to production to make sure that it's observable. And Observability 2.0 is about, it's the foundation of the software development lifecycle. It defines your velocity, how fast you can ship, how well you can ship, the quality of what you ship, your ability to iterate quickly, your ability to identify what your customers are actually doing and why, and build on that. It's your ability to see what's happening in the wild and make decisions based on real data and then feed them. Because this is all about feedback loops, right? And it's about learning to be a developer where you're developing with fast feedback loops.And it's like the difference, o11y 1.0 is about, okay, this is something that you tack onto a product...2.0 is about, this is how you build the product, right? So many teams are stuck in 1.0 land and they're happy with the tools that they have, but the teams that are going to win are the ones that not only adopt 2.0 tooling, but also adopt the 2.0 mindset of this is how we build software. It's like putting your glasses on before you drive down the highway. You can drive a lot faster, you can make better decisions much more quickly. So I feel like right now, the big problem that Honeycomb has from a business perspective is that far too few engineering leaders even understand that 2.0 is possible because you can have a 2.0 mindset. But if you've only ever seen 1.0 tools, it's janky. It's real hard to like...you can only do so much, right? You really need to see 2.0 tooling in order to really...But it clicks so fast when you do. So that's really our job. For a long time, I was really disappointed that there are still Observability startups starting. They come up, ping, pong, like here and there, everywhere, but they're all 1.0 tools. They're still doing the multiple storage places. My hope is, and I get why, it's because you have to build an entirely different storage layer from the ground up. And very few VCs have the patience for you to do that. They want you to get right to product, market fit and all this stuff. Now that there are more columnar storage engines out there like Snowflake, I don't know...I'm optimistic, but I'm optimistic over the long run, our model of Observability will win. Even if Honeycomb completely fucks up in the end state is the complexity of our systems is increasingly demanding it. The complexity of people's systems is skyrocketing. You look at the DORA metrics, and I was always kind of like, dude, it's so weird. Like high performing teams, okay, that takes an hour to a day to restore service. But for the bottom like 80% of teams, it takes them a day to a week to restore service from an outage. How? It's because they don't have Observability.It's because they can't actually see what's going on. They rely on a few people's brains, people who've been there for a long time, who pack a lot of context into their heads, who can try and reason about it using the very limited data sources that they have. That's why it takes so long over and over. Part of the reason we win so many of our POCs is because over and over, our sales engineers, we help you roll it out, and they'll be like, is this an outage over here? We're seeing something wrong. And people will be like, what? Ten minutes later they get paged and they're like, oh, it's just like once you have this feedback loop, you get used to being constant conversation with your code instead of just like shipping and waiting for someone to get paged. At some point in the next hour two year, right. It's all about hooking up this feedback eventually, even if it's ten years from now, the model that we're talking about is the shape that's going to win whether it's us or not because our systems simply demand it. There's no other way to build software at that kind of velocity and scale.ADRIANA: I completely agree and I think having that conversation where Observability is considered...is baked into like...you're shifting left on Observability basically, right? Were it's like...CHARITY: Exactly.ADRIANA: No, it's not the thing that's tacked on at the end per usual. It's the thing that your developers are considering in the beginning that your QAs are using to troubleshoot shit and write trace based tests and that now your SREs are like, "Oh, I've got the information to solve the problem!"CHARITY: So many of the promises of Agile development and all these SREs and all of these cultural movements, they've never really lived up to their full promise. And I feel like the reason is because it's not just a cultural thing. You have to have the tools that actually make hard problems easy as well. And the feedback loops with metrics and logs are just painful and arduous and relies on so much on manual cross-correlation and heroes jumping into the break. But when you have the right tools, you can just glance at it and see the answer. And it's what unlocks the ability of teams to just be constantly...When I think about modern software development, I think about feature flags which help you separate releases from deploy so you can be deploying small changes constantly.CHARITY: I think about future flags, I think about Observability, just the ability to see what the fuck is going on at any point. I think about testing in production and I think about, well, canarying. There was one other thing that was on my mind. There's really just a four thing and they all reinforce each other, right? One of them alone is okay, but you get all of them together. And it's a completely different profession than it is in software development, which is kind of still from the shrink wrap era. It's like you're building, if your world while you're building software is your IDE and your tests, that's shrink wrap days. Your world should be production and telemetry. You should spend more time in your production windows than in your IDE windows. That's what modern software development is like I think.ADRIANA: Yeah, absolutely. And the final point that I wanted to touch upon is you mentioning...having...the data that correlates right? Where you're not just having to figure out how it's stitched together. And tools like open telemetry definitely enable that. But then I guess part of the irony though, is that open telemetry allows you to correlate traces and logs and metrics. But then if your Observability backend doesn't have a way to show that correlation, then you're kind of up a creek too.CHARITY: So I am so glad that OTel came out when it did so that I think we were able to have a lot of influence on how the data is gathered. You're absolutely right. Part of observability is the presentation of the information. If you don't have the ability to slice and dice, if you don't have the ability to combine, if you don't have that single sort of truth, then you can't really reap the rewards of Observability, even if you captured it. But capturing it the right way is the first step, for sure.ADRIANA: Yes, absolutely. And so glad that OpenTelemetry has gone officially GA. The specification has gone GA end of 2023. Long time coming. I'm super stoked for that.CHARITY: It's a big moment in our industry.ADRIANA: Yeah, and I'm so glad also that so many of the vendors have come together to rally behind it. And it's really not someone trying to flex their muscles over everyone else. It's such a lovely community.CHARITY: The only lagger is Datadog. People need to keep putting a little bit of shame and pressure on them because they're the only ones who are not playing nice, but everyone else is, which is a tremendous achievement. Huge kudos to Splunk, who's got like 30 engineers working on integrations every day. We would not be where we are without Splunk.ADRIANA: Yeah, it's so great. It's so great seeing all these innovations, collaborations, and people really genuinely caring for the project.CHARITY: It's great.ADRIANA: And on that note, we have come up on time. And thank you so much Charity for coming on geeking out with me today. This was awesome. One item off the podcasting bucket list for me. Always a pleasure to chat with you. And everyone, please don't forget to subscribe, be sure to check out the show notes for additional resources, and connect with us and our guests on social media.CHARITY: Until next time, peace out and geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Vileela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout. My wonderful editor daughter will edit out any, any stuff. I pay her good money.CHARITY: How old is your kid?ADRIANA: She's 15.CHARITY: Nice.ADRIANA: That's a good age. Yeah. And she sports right now...she's sporting some really rad pink hair. Last year, she had gone purple, and I just took her to get a cartilage piercing, which I'm like, hey, I have no issue taking you. No issue taking you. I'll look away while it happens. Yeah, it's super fun. Super fun.CHARITY: I went to college when I was 15, and I felt very adult at the time. And now I look back and I'm like. I was a child. What was I doing?ADRIANA: You feel so old when you're in high school or like, when you're 15. I remember when I graduated college and I'm like, everyone looks like a baby.CHARITY: Yeah. Time of rapid change.ADRIANA: Yeah, for real.

About our guest:Kelsey Hightower has worn every hat possible throughout his career in tech, and enjoys leadership roles focused on making things happen and shipping software. Kelsey is a strong open source advocate focused on building simple tools that make people smile. When he is not slinging Go code, you can catch him giving technical workshops covering everything from programming to system administration.Find our guest on:X (Twitter)MastodonFind us on:All of our social channels are on bento.me/geekingoutAll of Adriana's social channels are on bento.me/adrianamvillelaShow Links:CompTIA A+ CertificationRube Goldberg MachineHerokuKornShellCapistranoCloud FoundrySpring BootDistributed denial of service (DDoS)HashiConfMitchell Hashimoto (HashiCorp co-founder)Armon Dadgar (HashiCorp co-founder)Borg whitepaperSidecar (Kubernetes)Nomad on Kubernetes (GitHub)Hashinetes Talk (HashiConf 2017)From Community to Customers (KubeCon EU Amsterdam 2023)ConfdFOSSDEM (conference)Apache License, version 2.0RAIDWestworld LoopAdditional Links:Kubernetes the Hard Way (GitHub)Transcript:ADRIANA: Hey, y'all, welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And today I have the pleasure of geeking out with me, Kelsey Hightower. Welcome, Kelsey.KELSEY: Happy to be here.ADRIANA: And where are you calling in from today?KELSEY: I'm in Washington state, so on the border of Portland, Oregon, and Washington.ADRIANA: Awesome. Well, let us get to it with the warm up questions. Are you ready?KELSEY: I am.ADRIANA: Okay, first question. Are you a lefty or a righty?KELSEY: Right handed.ADRIANA: All right. iPhone or Android?KELSEY: iPhone forever. And I've tried android. Given that I've worked at Google for almost eight years, I've tried, but I'm an iPhone person.ADRIANA: Yeah, I'm an iPhone person too. I never tried android. I went straight from BlackBerry to iPhone.KELSEY: I think BlackBerry was definitely...I was a BlackBerry person. I was also a Nokia person. But I think once iPhone really dialed in the ability to have third party apps in the App Store, iPhone all day.ADRIANA: Yeah, I'm the same way. That was like one big sticking point. And for us in Canada, when the iPhone first came out, we didn't even have access to the App Store. So if you wanted any apps, you had to jailbreak your iPhone until it finally became available...because we get everything a little bit late here.KELSEY: Awesome.ADRIANA: Okay, next question. What's your favorite programming language?KELSEY: The one that I can get things done in. So, at one point it was Bash, then it was Python, then it was Ruby when I worked at Puppet Labs, and then it's been Goblin, probably for the last ten years.ADRIANA: Cool. Awesome. And Mac, Linux, or Windows?KELSEY: Mac on my desktop. Linux on the server.ADRIANA: All right, next question. Dev or Ops?KELSEY: They're one and the same.ADRIANA: I love it. Okay. JSON or YAML?KELSEY: JSON. If I had to program against it, YAML if I had to write it.ADRIANA: By oh, yeah, I definitely agree. I do find, like, manipulating JSON in Python is nicer, but YAML is more readable.KELSEY: Yeah. To all the people that are like, JSON over YAML, let me watch you write it and see how fast you change your opinion.ADRIANA: Yes, I totally agree with you there. Okay, this one's a little more controversial, and you can thank one of my previous guests for hinting at it. Spaces or tabs?KELSEY: I don't care. I actually don't care if Python makes me uses Spaces and my IDE does the right thing. I'm totally fine, actually.ADRIANA: I'm down for that. Okay, two more questions. Do you prefer to consume video or text when you're consuming content?KELSEY: It depends. If I'm trying to learn, I need to read it, I need to see it, I need to be able to kind of backscan read it twice. But I do like video in terms of when people are really good at the human side of it. Right? Like, if they're expressing or showing me something, like, I want to see the code run. I want to see where they click. I want to see how they start. But when it's like learning something in the programming world, I need text. People are pretty bad at video and programming lessons.KELSEY: Like, oh, just write these three lines of code. I'm like, can you please scroll up so I can see what you imported to make this work? So when it comes to seeing code, I want to see no snippets. I want to see as much as possible, but if I'm just going through for the first time to get the flow, video.ADRIANA: I totally agree with you. And you landed on one of my big pet peeves. When consuming content for learning stuff, which is the code snippets, because I have been and I'm sorry, Hashi people, but this is a crime on the Hashi docs that I see all the time is that I get code snippets, and I don't get to see a full example on the site, and it drives me bananas. And I'm like, what does this apply to? Give me a full fledged code example? Link me to a GitHub repo at some point.KELSEY: I'm always asking, why are people writing docs out there giving me hints to a murder mystery?ADRIANA: Yes.KELSEY: Show me the whole thing. I don't need it to be cute. I don't need it to fit perfectly in your style guide. I just need to see the whole thing and what's going on. So I think people do it out of style. There's really no substance when I'm trying to learn.ADRIANA: Yeah, I completely agree. I do find it very frustrating. That's why, for me personally, whenever I do technical docs, I give excruciating detail. All right, final question. What is your superpower?KELSEY: My superpower? I think one thing that I've learned over the years when it comes to mentoring, specifically, I used to be all about sharing my expertise, my background, my learning. And I've noticed that I changed my approach to holding up a mirror in front of other people and convincing them to like what they see and the number of people who actually like what they end up seeing and follow up with me. I really felt like that is a superpower, that you can actually have that impact on people. So that would be my superpower.ADRIANA: That is such an incredible superpower. And I think it's so relevant to our industry, too, because we have a lot of smart people who suffer from impostor syndrome. And I think showing people that you are actually as good as you think you are is such a huge thing. Right? I mean, we've got some amazing stuff happening. I have some coworkers who are brilliant, and they're like, oh, my God, I feel like I'm just a hack. I'm like, Are you kidding me? I can't even keep up with some of the stuff that you're telling me right now.KELSEY: Yeah. And I try to get people to understand that sometimes you aren't as good as you want to be. And that's okay too, right? I think there's okay with making progress, entering to new domains, and just helping people just relieve the pressure. Ideally, if you're any good at this thing, you're going to always feel this way forever because you're humble enough to keep learning, so you shouldn't feel so bad about it.ADRIANA: Yeah, that's true. That's a very excellent point. So let's get into the meaty bits. One of the things that I wanted to share with our audience was how you came to be on the podcast. We met at KubeCon North America in Chicago this year, and you were doing a book signing. And I came, stood in line, the long line. It was totally worth it. And I was wearing this mask that had the sticker for the podcast, Geeking Out, and you said, "Oh, what is that?" And I said, "Oh, that's my podcast."ADRIANA: And you said, "Oh, I could be on your podcast." So I am so stoked that you were able to join. And yeah, I mean, I've admired your work from afar for many years. I find your approach to Kubernetes very accessible, especially because it's such a complex subject matter. So I wanted to start off with how did you get into this field in the first place? Where did you find your calling to make things technical things, gnarly technical things so accessible to folks?KELSEY: I want to answer that question, but I want to address this advice that I give to my former self and to people that I run into all the time. And they say, how do you put in the effort to make sure good things happen to you in your career and in your life in general? And so you at that book signing with the podcast on your mask. You're advertising to the world, this is my podcast, this is what I'm doing, and you're advertising what needs to happen next. And so for someone like me, I can see that clearly. I understand in that limited interaction that there is this opportunity that I could actually be on your podcast, because now I know you have one. I think a lot of people really confuse luck and that kind of effort, right? When you put that kind of effort forward, you tend to make things happen. And so I just want to highlight that part of you having that as part of your strategy of going to KubeCon, making the best use of your time and every human interaction. So kudos to you for doing that.KELSEY: But it's a perfect example of how people kind of design their own careers and create the world that they want. So that's perfect. Now to your question about this whole idea of explaining things simply to other people. When I was getting into tech, a lot of people come from various backgrounds I come from the...fast food was my only job background, and I didn't go to college. And so for me, learning technology was like a pivotal life-making decision. I need to get into this field. I admire people that are in this field.KELSEY: I don't know anyone that's in this field. And so I would go get all the books and just flip through them. I remember the first book I think I bought was the A+ certification guide. I was like, I'll start there. And you just go through all of this stuff and you look at all your notes, right? You're trying to simplify all the information to truly demonstrate that you understand it. And everyone knows that feeling of the A-ha! moment where you take something that is complex to you and you finally understand it, and your confidence level just goes up. It immediately goes up. And so that feeling, I've always enjoyed having that feeling because it felt so empowering.KELSEY: So whenever I had the opportunity to speak at a meetup, I've noticed that some people at meetups or conferences, they speak, and it's just like, overwhelming. Hey, here's this computer science diagram. Here's this map that you cannot understand what's happening, and they are happy with just leaving it as a mystery to everyone. And you're like, what the hell was that? You had this opportunity to let me have my light bulb moment, but you chose not to. You chose to try to overwhelm me with your vast understanding of things that I don't. And so I've tried to say, what if I can make people feel like I felt whenever I learned a new subject? So this is why I've always said, hey, now that I understand this thing, I want to show it to you as well. But before I can, I have to give you context where I came from, my understanding beforehand, and then what led me to that understanding. And then let me show it to you.KELSEY: And I try to use analogies and simple terms, and you can see the light bulb moment go off for people in the audience, and then it becomes a game changer for their own career. So for me, I think I got addicted to that. Like, hey, I don't want to talk. I don't want to write a tutorial if it doesn't have that impact on people.ADRIANA: I absolutely love that because I can completely relate to that feeling, the euphoria, the high that you get from solving a problem, especially something where you've had to really put on the detective skills hat and try extra, extra hard to solve it. So that is so wonderful. I love that so much, and I think it's so important because making learning accessible to people, I think, makes it fun too, because I agree with you. Like those gnarly architecture diagrams that just look overly complicated, and then your brain starts to wander, and then you miss some important thing, and then that's it your opportunity for learning. That thing is gone if you're watching that lecture, because it's just, like, way over your head. So I think that's so great. Such an awesome approach to really disseminating information across the industry, especially these are not easy topics to unravel, right? So, Kubernetes, for example, how did you come upon doing your work with Kubernetes?KELSEY: You know how you walk in on someone watching some hit TV show, they're on season six, right? And you ask them, what's going on? Why is this person not like this other person? They're like, I got to recap season three for you to understand what's going on on the screen right now. And so I think for a lot of people, Kubernetes was my season six, right? I had always been in tech trying to share information. If you would have caught me 15 years ago, you would have saw me at a Python conference teaching people about packaging Python applications. If you saw me maybe six years after that, I was at Puppet Labs trying to contribute to configuration management tools using Ruby. And so when I get to Kubernetes, there's a whole career behind me of trying to build similar systems without the terminology or the experience. You just know that there has to be a better way of doing things. So when I saw Kubernetes for the first time and really got hands on time with it, there was an a-ha! moment. I was like, you know what? All the scripts, tools, philosophies, techniques, it has now been serialized into this one checkpoint, and the industry has finally given it a name.KELSEY: And so when I got that feeling, you know what was next, right? It was like, hey, I can't wait to go to a meetup to show people this thing. And I think the reason why I was able to resonate with so many people is because I had that previous background of doing things manually, trying different automation tools. And so I was just so excited. Like, I think we finally found the thing we've been all trying to build, and it looks like this. And so I think a lot of people got to see that season. It was like, oh, he's the Kubernetes guy. But there's so much historical context that goes into why I was ready to have that conversation, make those contributions at that time.ADRIANA: That's basically the classic case of, like, everything you've done up until that point has led you to that moment, and now you're ready to take on that thing. Right?KELSEY: I became a better speaker than I had ever been prior. I became a better engineer than I had ever been prior. And I've gone through all of that experience, and I was able to really articulate what was important. And I think for a lot of people who have been on this DevOps journey for a decade, nothing is working. We're doing all of the things: CI/CD pipeline, infrastructure-as-code. We're missing something here. And I think the industry had overly focused on automation and not abstraction. And Kubernetes was that final thing that you could touch to say, there is a difference between automation and abstraction.KELSEY: And I think when people saw those new APIs, in many ways, I told people Kubernetes was like this type system to infrastructure. It was like a standard library that we'd never had. It's not like a thing that if you just install, it solves all your problems. But it's definitely a much better checkpoint than what people were doing before.ADRIANA: Yeah, absolutely. And it's one of those things where I feel like it's a bit of a love hate relationship with Kubernetes. Right. Because in some ways, it makes life so much easier, and then in other ways, it's like, oh, my God, this thing is so complex to try to unravel in your mind. Right.KELSEY: I want to address that a lot, because there are some people that think I am the biggest Kubernetes fan in the world, and I am not. I actually spent the last four years working on replacements. I spent so much time at Google Cloud working on serverless just to make Kubernetes go away. I learned everything about it because I think the best people that will replace it are the people who understand it the most. And the way I look at Kubernetes is very different. People look at it as a tool that is competing with their other favorite tool or some alternative ways of doing things. To me, Kubernetes is just another word in the dictionary, and my focus has always been, what does it mean? And as a contributor, what should it mean? And when I think about it as an aggregation of the previous ten year set of techniques, and you push them all together, you get this thing. And I study that thing for, like, wow, we've come a long way since those days.KELSEY: Also, you can see what's missing. And I think that part is where, for me, that's inspiring. Oh, this is what's missing. So this is where the opportunity space is. Go work there and solve that problem. But I think a lot of people get into, oh, this thing is too complex. And I always ask them, but do you understand it? If you don't understand it and you say it's complex, then I think that's a mislabeling of the situation. You can just say, I don't understand it, therefore, I don't know why I would use it.KELSEY: And I think that's a fair way to start the conversation. I think a lot of people are just dismissing it because it's complex, and I can do something much simpler, and then they tell me what they're doing. I'm like, that sounds like a Rube Goldberg machine. You just named 25 pieces of custom tooling so you can avoid using Kubernetes. I don't know if that adds up.ADRIANA: Yeah, that makes a lot of sense. I think it makes sense too, like what you said earlier about looking for something that could potentially replace Kubernetes, because I also think that we tend to get into this sort of rut where we think, well, it all ends with Kubernetes. But we all know that software has evolved so much in the last 20 years. Even everyone was talking about Heroku is this awesome thing, and now, yeah, Heroku is still in the picture, but other things have come and kind of taken our attention. So where are we moving towards then in this space?KELSEY: I think in some ways things haven't changed very much in 20 years. You write code, you build the code, and you try to do some process to get it on the server so people can use the code. About 20 years, people have been doing exactly that thing. Now, how people have gone about doing that thing, that's changed at different speeds. Some people are still writing KornShell scripts right now as we speak, deploying apps at their company, and it probably works well. Then you have some people that are still using tools like Capistrano because they want to use something that's written in their favorite programming language, in that case, Ruby. And so they just want to keep everything within their problem domain. And then you have some people who prefer platforms like Heroku, Cloud Foundry, you name it.KELSEY: I think the challenge has been is lots of people have been looking for that one solution for everything. I remember when Cloud Foundry, like the Heroku competitor that you could run yourself, it was like, look, twelve factor apps are the way to go, and you can write everything as long as you use Java and Spring Boot. You do that, you're done, you're great. And then it's like, okay, that's fine. What about my batch jobs? Where do I run those? Not there. What about my databases? Where do I run those? Not there? And then what happens is you end up having to bring in a second or third platform. And that's where the harsh reality of all of this stuff is, is that whenever we don't have one solution to solve everything, you end up having to complicate your infrastructure. And I think complicated infrastructure just the actual norm at this point.KELSEY: What the world wants in terms of if you have a public facing website, you're probably going to have a cache, you're going to have Cloudflare DDoS protection. Various security concerns that Kubernetes versus Heroku is such a small part of the decision making process that even if you got that layer right, it is such a small part of the equation that thinking that's where the complexity is, ignores the big picture, where I think things like Kubernetes are 1/100th of the equation.ADRIANA: Right. That makes a lot of sense. Now, on a similar vein of Kubernetes like product, you've also done some work with HashiCorp Nomad, right? How would you compare Kubernetes to Nomad for folks who aren't familiar with both?KELSEY: Respect to everyone that contributes. Because I've written lots of code myself, and you do the best you can. So we just got to make sure we get that out of the way. We're not attacking people here. So if you have a HashiCorp logo tattooed on your body or Kubernetes logo tattooed on your body, this is not about you at all. When I first saw Nomad, I remember, is when they announced it in Portland at one of the smaller first HashiConfs. And I was scheduled to give a talk about Kubernetes, and I changed the talk the night before to do Nomad versus Kubernetes. And I remember Mitchell, Armon and so many people from HashiCorp standing there watching the talk. Everyone's crowded in to watch the talk.KELSEY: And look, I'm not a mean person, so I'm not someone that's naively attack a project that I'm not working on. Doesn't make any sense. But I did learn it, got it installed. And the things I liked about Nomad, you got this single binary written in Golang. You just put it on the server and it's almost immediately ready to go, starting getting value, right? That part around, just go get a binary and just have it run on the server. It really, really made that easy. The part that wasn't great, though, is the API. You look at it and it's like, what is this thing? Right? I think I get it.KELSEY: And it felt like, oh, you're trying to copy the Borg white paper about what a task is, but you haven't used Borg enough to know that this is not what you want to copy. And so it was a good serialization of that knowledge that was out there. They built a very high performance fast scheduler. They optimized for scheduling, speed, and performance. But the thing I think that they missed was the ecosystem. This space now is about collaboration. So you have lots of people who want to build infrastructure, automation, tools. And the one problem we've had over time, in my opinion, is that you have to glue them all back together.KELSEY: And scripting only gets you so far when you have to glue together all these various APIs. So Kubernetes takes a different approach. Kubernetes says these things are all related. Your load balancer and your app and your IPs, and your storage, your secrets, all of it is related. And they depend on each other. And so Kubernetes felt like it lived a life where the maintainers or the people of that project had been using Borg for a decade or two and said, what would we fix? And they come into a popular ecosystem like Docker and all these pieces, and they aggregate them. And when you look at the API, you can see the experience peek through. Right here is a pod.KELSEY: A pod has to have multiple containers because most apps that people deploy in reality, need things like NGINX or sidecars or logging daemons. And so I felt like Kubernetes had so much more experience baked into it than just being a faster, easier to manage system for deploying things. So given that, it was really nice to see over time that both communities kind of learn from each other. I remember when Nomad started adding things like volumes, sidecars, or other things that you would typically see in Kubernetes. So I think some people like Nomad because of its simplicity. I kind of lean towards the simplicity side of the house, so I kind of resonate with the whole Nomad thing. But watching people kind of glue together, like vault console, and all these other pieces to try to get a whole system, I'm like, man, at this point, now Kubernetes starts to look a little better.ADRIANA: Yeah, I definitely agree. I worked at a job where so I had come from a Kubernetes background and worked at a job where it was a Hashi shop, and they're like, oh, we're using Nomad. So I'm like, oh, my God. How do I translate this? And when I learned that Nomad is not fully equal to Kubernetes, that you have to still stitch these other pieces together, I'm like, oh, okay, that complicates things. But I definitely agree with you. One of the things that I do appreciate about Nomad is that certain things seem a little bit simpler. And I did find the learning curve not too bad. Maybe it was because I also knew Kubernetes at the time, so maybe that helped and it allowed me to translate.ADRIANA: But there's definitely a lot of stuff that I appreciate about Nomad, and I'm glad that I've had exposure to both ways of doing things, because I think that's really cool. And like what you were saying, both communities learning from each other rather than, like, let's hoard our secrets, because that way you can end up with better products overall, right?KELSEY: 100%.ADRIANA: Now, one thing that I wanted to ask you about was your famous Hashinetes tutorial. What motivated you to put this together? And also, if you can just share with folks what this Hashinetes thing is.KELSEY: I remember the Hashinetes talk, because that was the year I was like, okay, all of these tools have been out for a while. Vault is out. Consul is out. Nomad is out. Kubernetes is out. Now what? How do you think about all these things? What do you even do with them? And I remember that year I wanted to have fun, right? Previous years, it's more about, what are these things? And then maybe years after that, it's like, it's in production. But I was like, you know what? I want to have a irresponsible talk. I remember starting to talk off: "Today we're going to be irresponsible."KELSEY: "Do not do this in production." "Do not go to work and say Kelsey said anything." This is just having fun. Okay, and so I remember having a Kubernetes cluster or maybe even Nomad, and said, all right, we're going to install Nomad as an app to see how it works. And I just started adding different layers and components one by one. Number one, teaching people how all of these things actually fit together and how another scheduler could actually arrange them and put them into place. And then I think people had so much fun with the talk. It's like, wow, look how powerful these tools are that they can actually deploy and manage each other if you really wanted to.KELSEY: And look how they're similar in some ways. And I think a lot of people were like, oh, these are just you need to pick one or the other. And at that time, there was a blog post of a company using Kubernetes for some stuff and then using Nomad for some of their batch jobs that would benefit from the Nomad way of doing things. I thought that was just, like, the right way to think about it. So that talk Hashinetes is like, what happens if you push Kubernetes and all the HashiCorp tools together, like using Vault for secrets instead of the thing that was built into Kubernetes, because I think Vault was a far superior secrets management product and API. And then what if you were to use Consul instead of Kubernetes built-in service-discovery? What would you get? And then let's just say you really do like Nomad. What if you were to run that inside of Nubernetes, too, and let that become the scheduler instead of Kubernetes doing the scheduling? And I think when people kind of saw that talk, they understood how to really fairly evaluate those tools. So we just had a bunch of fun.ADRIANA: What do you think was the biggest learning from putting this talk together for yourself?KELSEY: I think, honestly, if you just live 100% in Kubernetes land, all you know is config, maps, secrets, and you have an idea in your mind that there's no other way of thinking about these problems. Right? Everything must be a CRD. Kubernetes, Kubernetes, Kubernetes. But I think people forget I was a contributor to Kubernetes. I knew how some of the inner workings worked. And so it's like, how do you get Vault to work nicely inside of Kubernetes? Then you have to rethink the APIs, and you start, oh, the Kubernetes secret management API isn't that great at all? And so when you bring in Vault and you have to stitch it in and bake it into the whole process, you really do gain empathy for gluing all of these parts together yourself. So I think the biggest learning for me is that, number one, you can do it. There are situations where it does make sense.KELSEY: Think about it. If you have multiple clusters and you want to have multi cluster service discovery, you cannot do that with Kubernetes alone. When you add something like Consul, you can have Consul be the place that takes over DNS. And guess what? Voilà, you can now address multiple clusters using one service discovery tool. And so it's like, oh, okay. So even though Kubernetes hasn't solved all the problems, it doesn't mean that you can't bring in all these alternative tools to step in and fill that gap.ADRIANA: And it's nice to see that everything plays nice in that little ecosystem and that you can, I guess, take advantage of each tool superpowers, right, to sort of give that boost to Kubernetes Awesome. Now, on the Hashi front, I also wanted to talk to you briefly about a talk that you gave at KubeCon EU, "From Community to Customers". And I attended that talk, and I really enjoyed the talk. I thought it was very interesting how you were talking about this fine line of what to keep open source versus what not to keep open source. One example that you cited was HashiCorp, and then shortly thereafter, HashiCorp changed their licensing. So what are your thoughts around that?KELSEY: Yeah, I actually had this question come up a few times, and I always tell people from a place of empathy, I had a project, Confd. It became a little popular. I remember going to FOSSDEM on the other side of the world in Europe, and watching someone give a talk about using Confd, this miniature configuration management tool, and how they were using it and why they thought it was one of the greatest projects ever. Like, as a maintainer of an open source project, you'd love to see a community form around the thing you've built. But as a solo maintainer, you also know how hard it is to say no. And you wake up on, like, a Saturday morning and it's like, hey, I work at a huge company that makes tons of profit, and I get paid really well to do my job. I would like you to work for free and add this feature that we really, really need to make even more money. And you're like, no, this is not my priority.KELSEY: Number one, you're not paying me anything. And then two, you know what? You're going to have to prioritize that itself and maybe step up and do some contributions. And so when you think about it that way, and as someone who's also contributed code to HashiCorp products in the past, I did those contributions to scratch my own itch. And I understand that once I deliver those changes, it's on the HashiCorp team to maintain them forever. And so I understand the relationship here is me contributing code is not the end of the story. And so when they make that licensing change, I put myself in their shoes of trying to run a business and remember, they're a public traded company. So a lot of these decisions are not in fully their control anymore. The market wants to see profit growth.KELSEY: I don't know if you've ever worked at a profitable company, people listening to this. But having stagnant revenue year over year is a fast way to get shareholders to leave investing in your stock. So now they have this added pressure of no longer just making the open source community happy. The people that they kind of started their careers off of, now they have to try to make the market happy. And there you get into different behaviors. So now you got to figure out where to get revenue from. And if you ask someone, Where do you get revenue from something that is given away 100% for free? Last I checked, most people do not pay for things unless they have to pay for things. And so you got to draw the line somewhere.KELSEY: And I think the big controversy is, where do you draw the line? Do you draw the line on the core of the product? NGINX tried to do things like that. It didn't work out well over time, do we draw the line on, like, enterprise features and Web UIs? Right? That could be a fair place to draw the line. And so I think for a lot of people, HashiCorp decided to draw the line at commercial competition. If you take our software and start competing against us, using our name, likeness, whatever we say now in our new license, the business source license, that you can't do that. And so if you're being honest, as a user, don't really care. Like, I don't plan to start a business competing against terror. If you're being honest, I literally don't care.KELSEY: And most people don't really exercise all their open source freedoms anyway. I'm not saying that's not a good reason not to have them, but a lot of these licenses like Apache 2 to me to fully realize the benefits of them. I think you do need to become a contributor to really understand what the code base does, be willing to step up to fork a project when the time comes and having the skills to maintain it. A lot of people don't understand that's the other part of this deal. And so when they change that license, I think people got a wake up call. They own that project. It is not our project. Even those with that HashiCorp logo somewhere tattooed on their body, it's not your project.KELSEY: It belongs to HashiCorp. And so now I think there's a rethink. And a lot of people forget HashiCorp predates the CNCF, right? So they're not a part of a foundation, even though a lot of their technologies are foundational, TerraForm, Vault, those things belong to HashiCorp, a private company doing what they have to do. And so for me, I look at that business license change and says, great, they made their stake in the sand. From a business perspective, this will be good for HashiCorp. Now they can say no. And now their terms are a bit clear and no longer vague. Now, for the community that is upset,KELSEY: now it's time to exercise those open source rights we've all been talking about for so long. You get to fork the project, you get to maintain the project, bug, fixes security, fixes new features and then ask the question how compatible should you remain going forward with the thing in which you branch from? That's what's on the table. So those are my thoughts on it. It's very pragmatic. I think it's one of ownership and responsibility and no matter how you feel about it, you're going to have to take on ownership and responsibility going forward.ADRIANA: Yeah, absolutely. It makes so much sense and I think you hit on a very important thing when it comes to maintaining an open source project, which is maintaining it. It is a lot of freaking work and especially if it's something that you do on the side for funsies. You can only expect so much, especially if you're the solo maintainer. So also hats off to anyone who is a solo maintainer of an open source project or works with a very small team because it's a lot of work. It's a labor of love at that point, right?KELSEY: I want to make sure people understand. A lot of people may have an ops background. That's definitely where I come from. And people think dev is easy and there's the same stress that you have in operations, right. For example, if you replace a hard drive in a server with a bad hard drive, you worry the first couple of days like, is that RAID configuration going to actually rebuild on time and the hard drive is going to stop being slow before traffic comes. You worry about these things and this is why we started doing things like on-call. And when you are maintained of open source project, you know that anything you merge in will make its way to someone's production, someone you probably don't know and you're going to feel responsible and accountable for doing that. And so there's a lot of this added pressure of like, hey, I got to be able to say no and make the right decisions to make sure that no one is going to be negatively impacted by these projects.KELSEY: I think a lot of people forget that when we start to ask and I don't like the way this person runs this open source project, there is so much pressure that goes into it. So just know that there's humans behind these projects. There's a lot at stake. So if they say no to your new feature or they have to make a business license change or stop accepting pull requests for a while while they go tend to other matters, you just have to understand that just what comes with the territory.ADRIANA: Yeah, absolutely. There are humans behind those repos right at the end of the day. Well, we are coming up on time, but before we wrap up, I was wondering if there are any parting words of wisdom that you would like to share with our audience?KELSEY: I don't know if there's any parting words of wisdom, but I do think we're at this next cycle of new technology on its way, whether it's AI or LLMs, some people only know that stuff as chat GPT. And the question that I'm hearing a lot around is, like, is this thing going to take my job? And I always ask those folks, what is your job? And they say, "Well, for the last ten years, I've just been running scripts and automating things, and I'm like the same things for ten years in a row." I was like, "Listen, if that's how you would describe your job, then yes, you might have a problem when a new set of tooling comes around that reduces the need to do that." And that's always happened throughout tech. And I think what most people should probably think about is take these moments of insecurity and just do some self reflection and say, "Hey, my tools"...and I think we started the conversation this way. People tend to confuse automation to abstraction, and a lot of times, people get so comfortable automating the same things over and over, almost like a Westworld Loop, that they forget that we should rethink the thing that we're automating and ask ourselves if we should replace it with better abstractions. So I would say this this may be your very moment to pause for a second look at the work you do, and ask yourself, "Is it time for a new abstraction?" And if it is, I think that's the perfect opportunity to either go find a project that's attacking that problem or maybe even start your own that introduces the new abstraction based on all of that experience that you have.ADRIANA: Awesome. I really love that. Well, thank you so much, Kelsey, for geeking out with me today. Y'all don't forget to subscribe, and be sure to check the show notes for additional resources and to connect with us and our guests on social media. Until next time...KELSEY: All right, everyone, don't forget to Peace Out and Geek Out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout. Hey, hey Geeking Out fans! We're taking a little break for the holidays, so this will be the last episode of 2023. Be sure to catch us again in January as we Geek Out with a fabulous lineup of guests.ADRIANA: See you in 2024. And Peace Out, and Geek Out. Bye!

About our guest:Jacob Aronoff (he/him/his) is a Staff Engineer at ServiceNow Cloud Observability, formerly Lightstep, the tech lead for the Telemetry Pipeline team, and an OpenTelemetry maintainer for the OpenTelemetry Operator project. He's spent his career in a variety of backend roles acting as a distributed systems engineer, an SRE and a DevOps professional. Jacob's focus is enabling customers to reliably send telemetry data with a focus on Kubernetes and OpenTelemetry.Find our guest on:LinkedInX (Twitter)MastodonFind us on:All of our social channels are on bento.me/geekingoutAll of Adriana's social channels are on bento.me/adrianamvillelaShow Links:ElixirSwiftOpenTelemetry (OTel)OpenTelemetry OperatorPrometheusOTel CollectorOpenTelemetry Protocol (OTLP)OTel for KubernetesOTel Operator channel on CNCF SlackOTel End User Working GroupstatsdOpenTracingOpenCensusJaegerCommon Vulnerabilities and Exposures (CVE)OTel Operator Target AllocatorPrometheus Re-labelingOpen Agent Management Protocol (OpAMP)SignalFXAdditional Links:Adriana's articles on the OpenTelemetry OperatorJacob's Talk at KubeCon NA 2023Jacob on OTel Q&AJacob on OTel in PracticeJacob on the Maintainable PodcastAdriana on the Maintainable PodcastTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela. Coming to you from Toronto, Canada. And geeking out. With me today is Jacob Aronoff, who is also one of my coworkers. Welcome, Jacob.JACOB: Hello. Very happy to be here. I'm so happy that we get to do this. I feel like we talked about this in Amsterdam, and I'm so excited that we get to make it happen.ADRIANA: I know, right? Yeah. This is awesome. So as we start out, I'm going to do some lightning round questions. They are totally painless. No wrong answers. So are you ready?JACOB: I'm prepared. Let's do it.ADRIANA: Okay, cool. All right. Are you a lefty or a righty?JACOB: I am a righty. So I always thought I was supposed to be a lefty, and my parents forced me to be a righty.ADRIANA: Interesting. Soul of a lefty. iPhone or Android?JACOB: iPhone. I just got the new one. USB-C all the way.ADRIANA: I'm so jealous. I think I'm going to wait one more year because I want the iPhone...I don't like the Pro Max. It's too big. But I want the Pro.JACOB: It's way too big.ADRIANA: I want to wait until they upgrade the optical zoom to whatever the Pro Max offers.JACOB: Yeah, that makes sense.ADRIANA: Yeah. Anywho, go on. Okay. Mac, Linux, or Windows?JACOB: Mac for sure. Big Mac boy. Whole life.ADRIANA: Feel you. I feel you. Okay. Favorite programming language?JACOB: I feel like Go. I mean, I'm a huge fan of Go. It used to be Swift or Elixir. Those are my two a little bit more funky choices. I used to work in Elixir, and I really loved it. Definitely one of the most fun languages I've had the chance to do. Swift, I haven't done for a few years, but there are a lot of little Easter eggs around my socials that refer to Swift a lot.ADRIANA: That's why your social handle is get_sw1fty.JACOB: Exactly. Yeah.ADRIANA: Okay, I get it.JACOB: A lot of Easter eggs.ADRIANA: Nice.JACOB: Still, I was the first person to ever write a Datadog SDK in Swift, and it's still on their website.ADRIANA: Wow. That is awesome. Very nice. Very nice. Cool. Okay, next question. Dev or Ops?JACOB: That's a really hard one. Dev. I'm just going to say dev.ADRIANA: All right.JACOB: Ops is fun, but you're still doing Dev if you're doing Ops. You're still Deving. You're still Deving.ADRIANA: I like it. Especially modern Ops. Right? I mean, maybe not...well, even Bash scripting back in the day, right? Ops was more bashy, less like Terraforming.JACOB: Yeah. Back when Ops is mostly just like Jenkins scripting with Bash. That's still Dev. There's still a lot of Dev stuff in there, so it's always been like that. It's just new abstractions.ADRIANA: Yeah, fair enough. That's a really good point. I like it. Okay, next question. JSON or YAML?JACOB: It's just...I'm a YAML engineer. I can't deny it.ADRIANA: Yeah, I like YAML better. No disrespect to the JSON people out there, but I don't get it. YAML forces me to do indentations, but that's okay.JACOB: Yeah, that's all right.ADRIANA: Yeah, cool. Two more questions. Do you prefer to consume content through video or text?JACOB: Probably text. I love to read really long form things, especially, I don't know, I save a bunch of articles whenever I see them and they'll be like, ten minute, 20 minutes reads, and whenever I have some real free time, then I'll go through one or two of them and that is like my favorite way to consume. I probably consume more video, realistically.ADRIANA: Oh, really?JACOB: Yeah, I watch a lot of YouTube videos, like "How To" type things.ADRIANA: Yeah.JACOB: But I love to read more than I love to watch. Watching is too passive.ADRIANA: I get too yeah, I agree. I think that's what I find annoying about watching videos. Like, someone sends me a video link, I'm like, it better be like some short video. So if it's like an Instagram video or YouTube short, it's fine, but send me a five minute video, I'm like, I'm never going to watch it. Even if you tell me it's like the most wonderful thing in the world, I'm not going to watch it. I'm so sorry.JACOB: Or it's like, even if you watch it, you get so distracted by another thing. It's just like I don't know.ADRIANA: Yeah, I think the only way I can consume, quote unquote, a YouTube video is if it's audio only. So I'm like just doing chores around the house and listening to it, then it's okay, right? My brain is like it helps me focus better.JACOB: I feel that basically you're just podcasting at that point.ADRIANA: Yeah, exactly. Which I love me a good podcast.JACOB: Yeah.ADRIANA: Okay, final question. What is your superpower?JACOB: Superpower? I have a useless superpower. I can do a noise. I can make a noise that's really I can click with my tongue really loudly.ADRIANA: Okay, now you have to demonstrate.JACOB: I will, but it might disturb some people in this office. Okay.ADRIANA: Damn.JACOB: I don't know if that came through.ADRIANA: It came through okay over here.JACOB: It's really loud.JACOB: That was like a quieter one.JACOB: It's useful when it's like, I need to get someone's attention who knows that I can do that. And then I'll do the click, and then they'll be like, oh, there he is.ADRIANA: Nice. I like, that. Cool. All right, now we shall get to the meaty bits, which is sweet. Let's talk OTel.JACOB: Let's do it. I'm ready.ADRIANA: All right. Yeah. So I guess for starters you're involved as part of your so we both work at Lightstep, which I guess is now ServiceNow Cloud Observability. I guess you and I met because we both work in the OTel space, although we work in different areas of the OTel space. Why don't you tell folks what you do specifically around OTel?JACOB: Yeah, so I sort of got started with OTel two years ago when I joined the company working on the OTel Kubernetes story and what's going on there. Basically I came from a Prometheus shop that really heavily invested in Prometheus and I had sort of seen the great stuff with Prometheus and then some of the struggles with Prometheus and I came in and I was, you know, I now work on top of a metrics backend. What's the best way to get metrics there? OTel has the OTLP format and so I wanted to figure out the best way to get Prometheus metrics into the OTLP format and then into our backend, specifically in Kubernetes and what is the best way to do that. So sort of began this journey on the operator group, which is a SIG within OTel that works on a piece of OTel code that sits within your Kubernetes cluster, within your environment to make it really easy to deploy OTel Collectors and do auto instrumentation and things like that. And then the feature I was working on was to make it so that you could really easily scrape and scale metrics collection. So that was sort of my first foray into it. And then I started contributing a lot. I became a maintainer for the project and now I just sort of work on OTel Kubernetes stuff all the time. So thinking about new features, new ways to help users run their whole environment for telemetry collection in Kubernetes, that's really the focus.JACOB: How do we make that as easy as possible for people? There's definitely a lot to be done, but it's a really great group of people that I think think pretty deeply about this stuff and are very good at sharing and caring and not very what's the word? Nobody's really holding on to legos. Have you heard that phrase? Is that like a known phrase? Yeah.ADRIANA: I haven't heard that expression before, but I like it.JACOB: Everybody's happy to share. There's not really someone who's particularly unwilling to accept something. Yeah, nothing like that. It's really based on the merit of the feature, not the fact that you don't get to do it nice. It's a good group as a result.ADRIANA: I really like that and I can vouch for that too because I've bugged you with a bunch of questions around the operator when I was trying to understand it better. And I've also posed questions to the operator Slack Channel and people have just generally been really nice about answering my questions, which is awesome because I think definitely tech has, I would say. I'm sure it still exists. But you see stack overflows where people ask questions and then you get some asshole who's putting you down because you're a novice to the subject and you're just trying to understand it. I get none of that from the Otel community, which I love because then it makes me unafraid to ask questions and so it makes it easier to learn.JACOB: Yeah, and a thing that I try to make sure of, at least with our group, is for anybody who's like a new contributor. I try to go really out of my way to thank them for their contribution and make sure that they're sort of set up for success with what they're doing. Like, even today, someone was asking some questions on our GitHub about some operator features. I gave them their answers and they said, if you have more questions, reach out in our slack. Happy to follow up there. And so they followed up, asked some more questions. They asked for a feature that we didn't have. I was like, oh, if you make an issue for that, we can get that on the books.JACOB: It's not that hard. And then I was like, hey, this is actually really easy feature. If you wanted to contribute it, I can walk you through that process. I can show you an example of, like, here's an example that you can look at for someone who did something similar in the past and let me know if you have any questions. And that's what they're going to go do now. They're going to make their first contribution. So it's something that I'm really happy to see as not just with my group, but like, all the groups, people are really happy to walk you through contributions and make sure that you're supported. And if there's a feature that you want, people will actually take you seriously.JACOB: They respond to you with sincerity, not what's the other word? They respond to you with sincerity, not hostility. And so there are no questions that you could ask that I've seen where someone's going to really get angry at you for asking that question. And I think that that's, like, a really nice thing. It's good to see a humble bunch and not like, a really egotistical bunch.ADRIANA: Yeah, I completely agree. And I think that's why people keep contributing to OpenTelemetry, which is great. Now, as a follow up question related to OpenTelemetry, we had you on for the OTel End User Working Group for, well, two sessions. So first for our Q&A session and our OTel in Practice, which we host those two sessions on a monthly basis. And you had a really cool story, actually, about migrating to OTel within the context of an observability company migrating itself to OTel. And why don't you talk a little bit about that? I think it's so cool.JACOB: Yeah. So previously our company was on...before we had a metrics platform...we were on stated. Like, all of our metrics were recorded via statsd. Sometimes we would rewrite them in traces, which was pretty weird, or we would have them go through a proxy so that we could aggregate them in some way and get some information out of them. So we were previously on the statsd, and then we were also on a really old version of OpenTracing. This was before the OpenTracing and OpenCensus projects merged into OpenTelemetry. And so we were on that old OpenTracing version.JACOB: And so I took on this work to migrate us to OpenTelemetry for everything. Well, metrics and traces. Logs support is still in the works, but that's the next migration. But so I started this project for migrating our metrics to OpenTelemetry, at which point the metrics SDK was still in beta, or the metrics API was still in beta, the SDK was in alpha. And so the goal was to really help the people on the, you know, iterate on their designs, work on performance and really tighten up that spec. So I did that, and then I actually found a bug in our maybe not a bug, a performance issue in the metrics code, which was a result of us having to convert from the new OTel format for attributes into the old OpenTracing sorry, other way around to convert from the OpenTracing attributes format to the OpenTelemetry attributes format. The reason this was a problem was because we shared this implementation between our tracing and metrics, and it meant that every time we recorded a metric, we had to do this conversion on the fly. And it doesn't sound that bad on an individual basis, but when you're recording hundreds of thousands, millions of metric points, that's a lot of conversions and that type of thing can really add up totally. And after I gave some of this performance feedback to the team, I actually realized that we could do this OpenTelemetry migration for tracing as well, which would then get rid of this performance concern.JACOB: And so in the midst of the metrics migration, I took a pause and then we began the tracing migration. The tracing migration was much easier because it was a more mature format at the time. So that process was a bit smoother. There were a few weird things here and there. You can read about that, I think online somewhere that we have documented, maybe, I think there's some blog posts.ADRIANA: We have the recording from your OTel in Practice, OTel Q&A discussion as well.JACOB: Yeah, cool, thanks. But so we finished that migration, we went back to the metrics migration. We got to use that performance benefit. And the OTel people actually worked on a lot of the performance recommendations that we made. So we were able to finish the metrics migration as well. And so it was really neat because I love these types of migrations, because you're really just like, you'll see the phrase a lot, replacing the engine of a flying plane. It's like doing that in place. And that's really what it feels like sometimes when you're dealing with hundreds of thousands of data points per second, how do you replace your telemetry collection about that? That's a pretty challenging thing for any company, not just us.JACOB: But then when you're the vendor serving the metrics. It's like, who's watching the watcher? That type of thing. Really the most difficult part is just reorienting your brain to think about the environments correctly to be sure that when you're talking about environment A, you are sure that that's where the data should be and not somewhere else, right?ADRIANA: Yeah.JACOB: Because for most of these telemetry vendors, whether it's us or Datadog or New Relic, it doesn't really matter. All of them have a meta telemetry environment that's sort of the secondary place that they send the telemetry of their main environment to. So that's the thing that you're monitoring. That's what lets you do these migrations effectively as well.ADRIANA: Yeah. So here's a question because this is actually like a really cool use case, because when we talk about bringing in OpenTelemetry to an organization, if you're lucky and you're starting out your application from scratch, you have the luxury of factoring observability into your architecture, right? And so you can start instrumenting in OpenTelemetry right off the bat, hopefully, right? One can dream. But then you also have the so called brown field scenarios, right, where it's brownfield. I have zero instrumentation and then there's the brownfield of like, I have instrumentation, but it's out of date. And I think that's something or not out of date, but it's not up to date with a standard, which now like the standard being OpenTelemetry. And so those are two really interesting conversations to have because I think a lot of the organizations that are adopting OpenTelemetry probably fall into one of those two categories. And from talking to a lot of folks, it's interesting too, because you have this conversation of like, you start telling them, oh yeah, I work in OpenTelemetry. Oh yeah, OpenTracing, we use that.ADRIANA: And I'm like, no, not the same, not really. You're having to educate them on that. But folks are also like, even if you get them sold on, like, okay, OpenTelemetry is the thing you got to now talk about a strategy for bringing that into the organization. And that can be very tricky. I mean, where we're at, it was an easy sell because it's like, well.JACOB: Yeah, this is what we do, this is what we work on. We should be doing it ourselves.ADRIANA: Yeah, exactly. So that's not even the problem. But even with that easy...I'll say easy, right? Because you're not having to deal with that hurdle. You have the hurdle of like, well, I've got some existing stuff now that I have to migrate. So one thing I'm wondering is, as you mentioned, there was some old OpenTracing stuff in place. And one of the things about OpenTelemetry is that they say they're backwards compatible with OpenTracing, OpenCensus. Now, which from my understanding means that if you have that stuff in place, you don't have to gut it right away.ADRIANA: However, you probably don't want it to stay that way forever. So what do you say to folks who are in that position?JACOB: A real it's a benefit that OTel provides these bridges to these legacy formats so that you can start using OTel and then get all of that in place. The thing that I always think about whenever doing these migrations, whether it's like a service, your telemetry, it doesn't really matter. The question is, how long do you want to be in a dual state? How long do you want to be in a state where you're potentially confusing someone on call? It's like the real crux of the issue is it's like always imagine yourself on call for whatever service you're changing, and someone gets paged at, like, 3:00 A.m.. Do you really want someone to have to reason about where your telemetry is coming from or how it's getting generated? You don't you really want that to be consistent. You don't want to have to ask the question, oh, is this like an OpenTracing thing? Is this an OTel thing? In the same way that if you're migrating a service and you have legacy service and new service, if you're in the dual state for a long time and you get a page for an upstream thing that's related to both of these downstream services, it's really frustrating to have to ask the question, which of these downstream things is affecting me? Right? Yeah, it'd be much easier if it was just I look at the single downstream, and I know that's the problem. Basically, it's shaving the decision tree for.ADRIANA: This that you're doing.JACOB: And so anything that you can do to remove the amount of time that you're in that dual state, removing those branches is going to do you better in the long run. The migration path is good that you can do this. There's another path, which I also think is a great option, where the OTel Collector probably supports whatever format you have right now. I'd be surprised if it doesn't. What you could do is just send rather than installing a bridge into your code, you could just send your legacy format to the Collector and have the Collector output, and then you can change your application to use OTel in whatever time frame you want, and then just have that sent to the collector, which already accepts OTLP. Yeah, right. And so that'll help you actually verify that the migration worked. You're already getting OTLP.JACOB: You don't have to do anything with that. And then once you start sending OTLP from your application, you should see no difference in what's yeah, and that's a pretty verifiable thing. You could actually even use the file exporter on the OTel Collector to actually dump the data that you get. And then for Service A, run it with Jaeger for ten minutes, dump that data with the OTLP out, and then do Service A again, but with OTLP, dump that data for ten minutes, and then just see what it looks like, understand that you should see, like, a pretty minimal difference between those.ADRIANA: Right.JACOB: And that type of thing can give you so much confidence. And you can do that probably from your local environment without even needing to push it up. And so that's something that we didn't really consider as an option at the time. But had we thought of that, I definitely would have done it that way. It would have been a great option.ADRIANA: Yeah.JACOB: Where we could have just moved to OTel instantly and then backfill. Right. That's like a much easier path.ADRIANA: Yeah, I agree. I mean, it's a very low friction approach, especially at my old company. They were using OpenTracing in a few spots, and so the mention of moving to OTel kind of sent people in a panic. Like, we have to re-instrument. Yes, we do. But hopefully never again after. But that idea sent people in a panic, and I had the same thought as you, which was like, yeah, just pump it through the Collector. Like, you don't have to change your code right away, but with the intention of eventually changing your code.ADRIANA: Because now, correct me if I'm wrong, but if you continue on OpenTracing, you don't get to reap the benefits that you get with the whole OTel ecosystem, right? I mean, you don't end up with the traces and metrics correlation and the traces and logs correlation or any new updates to the API or SDK, right? You're kind of stuck with whatever OpenTracing was when it froze, when it was retired, basically.JACOB: Yeah. Which means if there are any CVEs, you're kind of like, out of luck. Which is a bad state to be in.ADRIANA: Totally.JACOB: It's a really bad state to be in.ADRIANA: Yeah. Awesome. Yeah, I definitely like that. Now, going back to the OTel Operator. So you said that you're doing mostly work around the metrics portion. It's the Target Allocator specifically, right?JACOB: That's exactly. Right.ADRIANA: Yeah.JACOB: Now it's a bit more than that.ADRIANA: Okay.JACOB: But back then, like, last year was basically all target allocator stuff.ADRIANA: Okay, cool.JACOB: I can explain it. So basically when we started this process, someone from AWS had designed this thing called the Target Allocator. The goal of it was that you could distribute Prometheus works in targets. Targets are things that are like IP addresses, like a pod, a node, your old EC2 instance, whatever it is. You then go and scrape that instance to generate metrics. Prometheus works where it's a single monolith and you have a list of targets and it scrapes those and stores that data. You have to do this because if you have more than one instance of Prometheus, there's no way to tell which instance should scrape which thing. And so you're just going to be duplicating those scrapes. With OTel, we have the benefit of we don't need to store those metrics because we're just handing them off to the next thing with OTLP.JACOB: So the Target Allocator's goal is to allow you to distribute those targets amongst a pool of collectors. So if you have 300 targets and you have three Collectors, the Target Allocator could say, I'm going to give each collector 100 targets evenly. Right, but you need to have 100.ADRIANA: Collectors then to send it to...is that what that means?JACOB: No, you would just have to have...sorry...if you have 300 targets and you have three Collectors, then it's 100 targets per collector and then you would just forward that to your destination. So it'd be like if your destination is Prometheus actually, which now accepts OTLP, you could have OTel do all of your scraping and then just send the data to Prometheus as your backend store, right? And that would be like a totally viable option.ADRIANA: Gotcha.JACOB: If you really wanted the ability to shard your scraping and scale how you scrape targets, that would be a pretty viable approach.ADRIANA: Right, which Prometheus doesn't support the sharding right now, right?JACOB: So Prometheus has experimental sharding support but it doesn't have the ability. So it can shard your scraping, but it can't figure out your querying effectively. So because Prometheus is also a database. If you have three instances of Prometheus that are scraping each different targets, you'll only be able to query...you'll have to query the right instance each time because it doesn't know how to do that communication...to ask for, "Who has this metric?" At least that's my understanding of it. Maybe they've changed that, but I don't think they have.ADRIANA: Cool, okay. Yeah, that's super interesting. And so this allows you to scrape the Prometheus metrics which are not I mean, basically you're scraping it from wherever your source of Prometheus metrics is, right? It can be whatever, it can be coming from your infrastructure or whatever. And then this thing basically does the sharding for you and then it'll send your metrics to a destination. The destination could be Prometheus itself or it could be any observability backend that supports metrics essentially.JACOB: Yeah, yeah, exactly. Cool. And that's the real benefit. I mean, we also open up by using the Target Allocator, we can be a little bit smarter as well. So the thing that Prometheus does, because it's all in one, is most of the targets that you get, you're just going to drop. The way that the scrape configs work is you get a target which has a bunch of metadata and then your scrape config determines whether or not you should actually get the data from that target.ADRIANA: Got it.JACOB: Even prior to making the request. And so usually you have to keep all of those in memory because you're constantly scraping them and you're constantly asking this question does the metadata match my scrape config? Does the metadata match my scrape config? And so forth. Whereas because we have the Target Allocator, we can actually just drop any targets that we know the Collector won't scrape okay in advance. So we only tell the Collector to process targets that it will end up scraping.ADRIANA: Okay, so it's like a filter.JACOB: Exactly. That's what we call it. We call it a relabel filter.ADRIANA: Okay.JACOB: So the real reason that this is really cool and why we added this in is because then we can also really evenly distribute targets to Collectors because we can say only. So if you have 300 targets, we use this strategy called consistent hashing, where you just hash each target and their metadata to assign that to a Collector ID. And so if you have, like, let's say, 500 targets, but you really are only going to end up scraping 100 of them after this filter, it would be better if you only tell the Collectors...if you only distribute the targets that you're going to end up scraping, because it's going to be more even rather than trying to fit in. It's the pigeonhole principle, right?ADRIANA: Yeah.JACOB: If you have three boxes and you have 500 targets, you might evenly distribute it at first, but eventually, when you go to scrape them, it might be uneven once you figure out what you're actually going to scrape.ADRIANA: Right. By the time the Collector is receiving them, you've already just gotten the ones that you want, and so it can give you an even distribution of those. So then there isn't an imbalance, basically.JACOB: Yeah, exactly.ADRIANA: Nice. That is super cool.JACOB: It's very clever.ADRIANA: Every day. Yeah, that's very awesome. So is the Target Allocator only part of the OTel Operator? Is that something that's available as part of the standalone collector?JACOB: So the Target Allocator is its own image. Like, it runs separate from the Collector binary. You could theoretically run it without the Operator. There are definitely some people that do that, but we don't support that as like, first class support. Reason why is that we do a lot of logic to rewrite. In order to make this work, you have to rewrite the Collector's configuration, and you also have to rewrite the Target Allocators configuration. It's just a bit of, like, data munging that we don't want users to have to do just because it's a little bit complicated. So we do it in the Operator for you.ADRIANA: Yeah.JACOB: There are people who will take what the Operator gives you, remove the Operator, and then just run it themselves.ADRIANA: Right.JACOB: And that's kind of a viable option. Yeah, but that's bespoke you'd have to do that yourself. And if you ask me a bunch of questions, I'll try to help you, but there's a certain point at which I can't help you. I don't know what you're doing.ADRIANA: That sounds like someone's idea of, like, a fun weekend project.JACOB: So we have a bunch of requests from people to enable the Target Allocator as part of the Helm chart, the raw Collector Helm chart. And I tried to do it, and it was so hard. It just proved so difficult to do. The config rewriting was so challenging because Helm isn't really a language. It gives you some go templating stuff, but at a certain point, it doesn't get you all the way there.ADRIANA: Right.JACOB: And so I wasn't able to make it work, and I eventually decided to give up because it was too much of a time.ADRIANA: Yeah, that makes sense.JACOB: Which is unfortunate because people ask for it a lot.ADRIANA: Yeah, that's interesting.JACOB: Yeah.ADRIANA: Now, obviously there's an OTel Operator because obviously a lot of people run the Collector in Kubernetes. Do you know, is it common for people to run collectors outside of Kubernetes? I mean, obviously, if you're not a Kubernetes shop, I would imagine that would be the use case. But how common is it? Do you know?JACOB: I don't know. I mean, I'm sure there are a bunch of people that do it, because I'm in my little Kubernetes world, I don't hear about it that often.ADRIANA: Yeah, fair enough. Fair enough.JACOB: I'm pretty isolated, but there are definitely people who just run Collectors as binaries on raw EC2 instances.ADRIANA: Yeah.JACOB: GCS instances. People are doing it, for sure.ADRIANA: Yeah.JACOB: I don't know. They probably have a whole different class of problems than the one.ADRIANA: I know we're coming up on time, but I wanted to ask you quickly. Well, by the time this episode comes out, I don't know if KubeCon will have passed, but all the same, but do you have anything coming up at KubeCon that you want to talk about?JACOB: I do indeed. So one of the main projects I'm doing for the Operator right now is adding support for the OpAMP protocol, which is a new part of OpenTelemetry that gives users the ability to do remote configuration management and agent configuration and Observability, sort of, with superpowers. And I'll be giving a talk with Andy Keller from ObserveIQ on OpAMP and how it's going to make your life a lot easier to manage these pools of Collectors that you have. So I am working on this project in the Operator group that will allow you to basically understand the topology of your Collectors in your Kubernetes cluster and also remotely configure them. Add in new features, push out updates, everything that basically allow your cluster's observability to be on autopilot for you.ADRIANA: Nice. Who doesn't love that? Very cool.JACOB: Stop thinking about it.ADRIANA: Is that part of Observability Day, or is that part of the KubeCon, like the main conference?JACOB: Main conference.ADRIANA: Nice. Very nice. Yeah, very cool.JACOB: I don't know how many people can fit in the room that I'm in, though. I thought they'd tell you that, but I guess they don't.ADRIANA: It'll be a surprise the day of.JACOB: It will. It'll be anywhere from five people to 500 people.ADRIANA: I'm always nervous for these types of things. I think on the KubeCon schedule, you can see people already will sign up for your talk and you start seeing people signing up to attend your talk. And if it's like a small number, you're like, oh my God. And if it's a large number, you're also like, oh my God.JACOB: Yeah, I'm very nervous. Yeah.ADRIANA: Is like a very big deal. But yeah, this is awesome. Very excited for your talk. Oh, the other thing that I wanted to mention also, I don't know if it's going to come out by the time this comes out, but I do want to promote it because you were on the Maintainable podcast, you recorded an episode recently.JACOB: I did indeed. I don't think that's out yet, but definitely something to look out for, though I have no idea when that'll be out.ADRIANA: We will find out. Yeah, I think when I recorded an episode, I want to say like, in the spring and it came out a couple of months later.JACOB: So probably there's a backlog of editing.ADRIANA: Yeah, exactly.JACOB: It's a whole process.ADRIANA: I feel you. I have a backlog of editing for this too.JACOB: Yeah, that's just how it happens.ADRIANA: Yeah, totally. But anyway, something to look forward to as well, so you all keep an eye out for that. Now, before we part ways, do you have any interesting pieces of advice, be it like in tech or OTel or whatever, or any hot takes that you wanted to share with folks?JACOB: I think the thing that I always say is just do something that you enjoy. If you're looking for a job, just like find something that work on a project that you enjoy. Find something that's weird and fun and doesn't really matter and just brings you some joy. I think that we all sort of forget that coding can be really fun and enjoyable and there's so many things out there that are so cool right now, especially. And there's so many things that I think have been forgotten just out of the consciousness. I used to do a lot of coding with SignalFX and Java to do UI building and games and stuff, and I haven't done that in so long, but I had so much fun doing that. So if you're looking for a job and you don't know how to do it, my best advice is to do a project that you find very fun and interesting and not just one that you think will play well on a résumé. Because if I'm interviewing you and you tell me about a project that you were so happy to do and really excited about, that's going to be ten times better than a project that you didn't really care about.JACOB: Yeah, just have fun is my advice.ADRIANA: Yeah, that is really great advice and I couldn't agree more. Yeah, and coding should be fun. It definitely puts me in a happy place when I'm working on an exciting project that I dream up some weird thing that I want to explore and then you learn so much and I don't know, you get a high. The programmer's high.JACOB: Exactly.ADRIANA: Totally down for that. Awesome. Cool. Well, thanks so much, Jacob, for joining today. So y'all, don't forget subscribe. Be sure to check the show notes for additional resources and to connect with us and with our guests on social media. Until next time...JACOB: Peace out and Geek out.ADRIANA: Geeking out is hosted and produced by me, Adriana Vileela. I also compose and perform the theme music on my trusty clarinet. Geeking out is also by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to Bento Me slash Geeking Out.