Geeking Out with Adriana Villela

About our guest:Rob Golabek is Chief Architect & CEO at Translucent Computing. A thought leader and insightful tech visionary with over 20 years of experience, Rob is a Cloud Native expert specializing in App Modernization. Leveraging data, AI & cloud for digital transformation, he provides expert guidance to clients navigating the complex, ever-changing cloud-native landscape.Rob has shaped the technology landscape through his work at Translucent, progressing from software development to architecture and leadership roles. His expertise in cloud-native technologies, DevOps practices, infrastructure tooling, and tailored consulting approach helps clients drive toward cloud-native success, including observability and robust cloud foundation building.Rob also leads the ExecutiveEspresso Series, where he contributes to fueling business growth and inspiring the next generation of innovation.Find our guest on:LinkedInX (Twitter)Find us on:All of our social channels are on bento.me/geekingoutAdriana’s X (Twitter)Adriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:Translucent ComputingEtobicokeLefty DayJavaSickKids HospitalMetavante/GHRCanadian Army ReservePinky and the BrainPoint Click CareExecutive EspressoCloud NativeAngularTekStackFalcoKeyCloakPostgreSQLRedisOpenSearchKafkaPrometheusLokiLogz.ioKratixNatural Language Understanding (NLU)Neuro-Linguistic Programming (NLP)Natural Language Generation (NLG)Large Language Model (LLM)AI OpsDALL-EcapybaraJava enterprise serverTucowsNomadVMWare TanzuOpenStackAzure StackBank of MontrealCanadian TireHashiCorpVault (HashiCorp)Consul (HashiCorp)Jaeger (tracing tool)KubernetesTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, Reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And geeking out with me today is my good friend Robert Golabek. Welcome, Rob!ROB: Hey, nice to be here.ADRIANA: Super nice to have you on. And full disclosure, Rob and I have known each other for a really long time, like since, what...2000...I want to say 2005? It's been a while. We've known each other for a really long time in a past life, in our past lives as Java developers, which is really awesome. So Rob, for starters, where are you calling from?ROB: I am from the deep west Toronto Etobicoke.ADRIANA: Yay. Fellow Canadians.ROB: Yeah, people don't know Etobicoke is a borough of Toronto, so some people call it Toronto, some people don't. For some people, I heard it's really far. For me, it's actually the perfect balance. Twenty minutes from Toronto. But yeah, get some kind of space. So yeah, that's kind of where I'm from.ADRIANA: Cool. Awesome. Awesome. All right, so we're going to start with some rapid fire questions. Are you ready? I promise it won't hurt. All right, number one, are you a lefty or a righty?ROB: Righty. And happy Lefty Day. I saw that post, so yes, your superpower...I was going to respond post of my right-handed rights.ADRIANA: I always forget to acknowledge Left-Handed Day. And then this year I'm like, "I am going to schedule this post so I don't forget." And then when it popped up the next day, like on Monday when I was back at the office, I'm like, "Oh, Lefty Day passed. Oh, I remembered post on that."ROB: It so the reason it's close to me is my dad is left-handed, right?ADRIANA: Awesome.ROB: And for some weird reason, it was weird when he was growing up to be left-handed. So they tried to even make him write it with the right hand, and it was kind of, you know, so yeah, it's dear to me.ADRIANA: Yeah, totally. Yeah, my mom too, she was left-handed and she was subjected to people trying to make her write with her right hand. And she was one of those non-functioning-with-her-right-hand lefties...everything with the left hand. So she's like, "No." I can manage with some right handed stuff, but lefty and proud. All right, next question. iPhone or Android?ROB: AndroidADRIANA: All right. Mac, Linux, or Windows for development?ROB: Windows.ADRIANA: Awesome. Favorite programming language?ROB: The one I know, I got to say JavaADRIANA: All right. Dev or Ops?ROB: DevOpsADRIANA: Yeah, I've gotten a few of those answers before. It's very PC. DevOps. All right. JSON or YAML?ROB: Depends on the situation.ADRIANA: All right, fair enough.ROB: All right.ADRIANA: Fair enough. All right. And then final question: do you prefer to consume content through video or text?ROB: Text.ADRIANA: All right. Yeah, the text people are winning so far. Most people are like, "Text." I'm right there with you.ROB: And if there was a video, I watch it on mute. I like the writing.ADRIANA: Do you read the subtitles?ROB: Yeah...I don't know, I'm not a video person.ADRIANA: Yeah, I know, right?ROB: So kind of my age, I guess.ADRIANA: My daughter Hannah, she's like video, no question about it. I'm like, really?ROB: Yeah, I'm the same way.ADRIANA: But yeah, maybe it is an age thing. I don't mind...I'll watch video with subtitles or I will just put on the audio and walk around the house and have it on YouTube...video on my phone, walk around the house with just the audio, and that I can consume...but I can't just sit there and watch a video. Especially for tech stuff.ROB: Yeah, my my attention span is like, really, like, short. I want to kind of go to the end ofthe video, and I just want to read it very quickly because I usually skim through it and then I read the most interesting part in video. It's like, okay, where's the climax? You can't really find it.ADRIANA: Yeah, I'm exactly the same, so I totally feel you. All right, so now that we're warmed up, let's geek out on some stuff. So I guess first things first. So why don't you share with everyone what you do? Because you've come, I guess, a long way from our early days in our earlier careers of being the lowly Java devs.ROB: So yeah, so maybe start from the beginning, you know? '96, '97, '95. I don't know, just kind of coding. And at that point was involved with wires and illegal streaming. Got me interested. Kind of made some money from there. Very quickly. Went to Sheridan in 2000. Within a year, kind of graduated and then got my first job at SickKids Hospital as a reports developer that turned into Java developer, that kind of turned into architecture. It was pretty cool. And after that, that's when I went to Metavante, or is that the right name? I don't know. I think...ADRIANA: I don't know what they're called anymore because it was like when I joined, it was called GHR, and then Metavante ate them up and then I don't know what happened after that.ROB: Yeah. So I don't know. In between that, I was kind of in the Canadian Army Reserves too. So kind of got some discipline there. So yeah, it's kind of put me straight as an arrow. Got me kind of healthy and got some responsibility skills. That was from like '99 to 2004 while I was at SickKids. And then between '99 and 2010 while I was still working, I had a side business. So funny story, is in my first resume that I submitted to, I had like, a quote where I want to have a worldwide business where I kind of want to dominate and kind of provide value to people. And when I gave the resume to the SickKids people, they laughed. Right. And I'm like, was that naïve or was that aspiration to kind of something greater. So the entrepreneurship was always there, looking backwards. Maybe a little naïve but kind of inspiring to something greater was kind of my goal. That was kind of my beginnings of trying to take over the world. Pinky and the Brain 2006 that's kind of where I met you in Metavante. Worked there for three years. Went to Point Click Care. I don't know, I think everybody kind of knows here in Canada. Point Click Care one of the kind of unicorns in healthcare.So I was there for six months. Sad story is I joined, somebody got the bonus for referring me - it was my brother - who's kind of with the company as me and then six months later I leftADRIANA: Right when he got his bonus I'll bet.ROB: And they changed the rule after me. I think they even call it the Rob Rule that referral...you got to work there a little bit longer. I didn't do it on purpose. That's kind of when I started my business after Point Click Care. Got my first contract kind of working and actually was with SickKids too, developing their platform and that's kind of where my journey started. And we're here today. And what we do is right now we matured and kind of through the innovation that we do and putting engineering before sales, which I don't always advise because if you have passion for engineering and you want to do everything right, it might hurt sales. But we're proud of that. We run the business our way. So because of that we always kind of innovate, not always to the benefit of kind of sales, but it got us to the journey of early adaptation of Docker, Kubernetes, Cloud Native always early adapters and now we're Cloud Native experts specializing in app modernization, trying to kind of build for the Cloud and the beauty of Cloud Native and optimization, which I love, is it's ever changing, right? So before was moving to the Cloud was legacy software. Now it's kind of the hot take is how do you add AI to software that already kind of are out there, right? In a few years it's going to be something else. So really love what I do, kind of giving the Cloud Native expertise and kind of sharing my wisdom with people.And through that, sorry, I started Executive Espresso series where I started kind of like, you know what I love kind of talking to people. So started posting information, just kind of sharing on Cloud Native expertise and kind of the different aspects...Kubernetes, Observability...one thing that's challenging, which I tell you and it hurts me, is to be Cloud Native expert. I keep reminding myself how big the space is and like DevOps, Observability, Platform Engineering and cloud foundations, it takes a lot of learning and knowing and talking to people like you and different spaces. So I find that really challenging. But I enjoy that because in my DNA it's kind of learning. So combining all those things is pretty cool.ADRIANA: And you touched on something really important, which is like the Cloud Native space is ginormous and technology is ginormous and there's a new thing out all the time, so then you can't stay on top of everything. So how do you pick what you focus on as a result of that?ROB: So we can bring up maybe when you're doing the edits, you can bring up the landscape of the Cloud Native landscape. And I don't know how many tools they have now. Maybe 200, 300, a lot. So what we focus on is opinated experience technologies that we use. So we call it our Tek Stack, kind of powered by open source software. And we chose some tools right as the starting point. Now when we go to clients and kind of try to kind of give our opinions, it's based on that. Now it's also being open to other tools. But when you choose a tool, let it be mature, let it be kind of used by people, let it be a supporting community.We did a mistake before in the past, where we were too early of an adapter and you pay the price. I think we did it with Angular 2. We did it way too fast. When Angular kind of went through versions of one to two was Angular 1, then it was 2, then it jumped all the way to 5, was too early. I wish we waited a little bit and kind of used it maybe a little bit later. And same with these tools.So we broke it down into different tools for security. It's Falco, Consul, Vault, KeyCloak, kind of maybe HashiCorp kind of world. And then for kind of cluster resources, Postgres, Redis, OpenSearch, Kafka.So you can see it's like main kind of tools that we kind of use.And Observability: Prometheus stack...Kubernetes Prometheus stack, Sentry, Jaeger, Loki...Kind of making sure that we center on those tools and then making sure that adding principal infrastructure as code kind of on top of that and on top of Google, that's kind of how we chose the tools.And that's like the starting point, right. You can see for Observability, I think it's a very similar stack as Logz.io uses or anybody kind of those seem to be the main kind of open source tools that are out there, and there's a lot of support for them. So that's kind of the biggest kind of aspect of selecting them. And they're really good, right?So, yeah, that's kind of how we use them. But the biggest thing is through clients and through conversations, you always learn about the new tools. So best way is to throw your tools out there and then tell you some.The conference you went to, I think, was from you. You threw one tool, I forget the name of it, that I never knew about. And I was like, okay, it was for platform engineering, I think, or I can't remember which tool it was. But you were using your presentation. It'll come to me.ADRIANA: Oh, yeah. Kratix.ROB: Kratix, right, right. I never knew that tool before because I never came across it. Right. But then you use that and then it kind of opens up, and then I can query you and be like, hey, how do you use it? Where's the support? So learning from the community and kind of expanding it and then making a selection, hey, is the tool that we want to use or not? Do we want to add it to our stack? Right. So that's pretty cool.And I'll finish with this. That's kind of where the new thing of platform engineering is, I think. How can we best, to your question, select the best tools that maybe if I was going to propose to you, but also switch the game, what are you comfortable with and building around that most important part?ADRIANA: Yeah, that's so true. It's so important because there's nothing worse...and I've been in the consultancy space before...and there's nothing worse than coming in and saying your stuff sucks and then you're just going to hurt their feelings. It's like basically saying you have an ugly baby and no one wants to hear that they have an ugly baby. You got to be gentle and understand. What are you comfortable with, what are you using? Hey, would you be open to switching over to this? If you're familiar with this, maybe this might be the thing for you. And I think that's very important, especially in consultancy, because you're essentially trying to help companies do things better. But there can be a lot of resistance to change, so you have to be very gentle with them.ROB: It yeah, I don't know if I'm an engineer anymore. I'm ex engineer. I love engineering, but I spend more time doing non-engineering stuff. But there's one thing, right, that I always notice with engineers kind of myself, too, not excluding myself. There's that ego, right? I selected, I know the tools better. Prove me wrong. Why are you using this tool? And I don't like taking that conversation there. I'd rather being like, hey, if it's tools great, let's use it, let's improve it. Let's build what you guys need.Right? But engineers are smart people. I'm going to say that they're usually intellectually smart, so they know what they're talking about. And you got to come with a game, too, to say that you know what you're talking about. So that's kind of where the conversation goes.ADRIANA: Yes, absolutely. Is definitely a fine line. And I think one of the things in engineering, that engineering is an art form, really. And I think that goes to say for any type of art form is that sometimes we tend to fall in love with our code, with our technology, with the things that we create. But the best thing that we can do for our art is to give it some sort of a seed so that it can grow, whether it's like, hey, that sparks another idea where someone's like, hey, you know what? You could do this a little bit better. I like where you started, but I think this is how it can be improved. And being able to let go of your initial notions and be open-minded to other ideas, other ways of improving it, honestly, I think that's what open source is all about and I think that's what makes also for very successful organizations and very successful teams that you have to check your ego at the door.It's hard though, because sometimes you're working on a thing and it's like it's your baby. You've put a lot of TLC into it only to have someone say, well, I found a better way of doing it. And it pretty much scraps all the stuff that you did that can hurt. But also recognizing that maybe your initial work, even though it's being discarded, inspired somebody to come up with a better way of doing things.ROB: Yeah. I was going to ask you, what do you think is the best way of judging that? Right. How do you best put it out there? You kind of answered, I guess, open source. Right. Kind of let the community play with it. Any other kind of ways you would kind of try it out. Kind of let kind of people give you opinions in a non hateful control fashion.ADRIANA: Yeah, generally just having conversations. I think it all comes back to community, whether it's putting it out there through open source or writing about it in a blog post or having a conversation with somebody. Finding ways to make those connections, I think is probably the best way but you can't do that without it being out there in some form or another, I think.ROB: Yeah, I really like kind of in my recent time writing, right. So got me thinking and expressing and talking to people. Right. And then the biggest thing is taking that feedback in a positive way.ADRIANA: Yeah.ROB: First reactions like, oh, man, why did he say it that way? But then it's like, why did he say it that way? Maybe explore that a little bit more. Right. And then you meet the person, and then you have a different kind of perspective, and then you can change or you don't have to change. The biggest thing is to agree with themADRIANA: Yeah, absolutely. But I think the most important thing is that someone offering an opinion forces you to take a step back and rethink it. And it's like what you said, I'll either agree or, hey, there's something to that statement. Maybe I'll tweak it or take some of that into consideration, or like, no, I actually think my way is the better way. I've given it some thought and that's perfectly all right.ROB: Yeah. And you might have different motives too, right. It could be business case, could be technology, could be different case. Just recently, we had somebody come in and they had an objective of looking from this kind of zoom...was monetary zoom. And it's like that's one way of looking at it. Right. And because it's a business client, they're going to push it in that way. Now, as an engineer, the most frustrating part is let go of your best practices. And then because most of the times client is right. Quote. You try to kind of make them happy but you got to really put your ego away and also put away I told you so because I believe even in that scenario business person could be right because now they're coming from their perspective with and they might have limits.So you got to look at from that angle ego from technology, from business and kind of move the conversation forward.ADRIANA: That makes a lot of sense. I think at the end of the day, you just have to be open-minded. So of all the technologies that you've been working with, what's the one that's really exciting you right now?ROB: I love Conversational AIADRIANA: Oh, yeah. Cool.ROB: So...and applying it to any domain. We're just working actually with Pat, working on Conversational Kube Bot, where you can talk to it in human language and get a response.ADRIANA: Oh, nice. Is that something you guys are developing?ROB: Yeah, we're developing it. I want to release it. It's kind of started as a kind of small project because we're in a grander schemes working on Enterprise search, and we call it Conversational Enterprise Search, and we call it, like, Next Knowledge Base Economy, where knowledge is king. And how can you take that knowledge and how can you converse with it right. At a basic level. Right. And applying it in Kubot, hey, get all the resources, all the material from kind of Kube Bot and then suck it in. Use kind of NLU, NLP, NLG, kind of all the kind of natural processing, human and language kind of processing. So you're able to create something where it's your human assistant. Right. So my goal is, like, I never want to remember a Kubernetes command. And with this, we already have a prototype where it's like, hey, tell me the status of the system and let's see all the pods or something, right?ADRIANA: Oh, my God, that's so cool. I cannot tell you how many times, if I'm away from Kubernetes for a while, I have to Google this stuff. Or now I have a GitHub repo where I just have a README with all of my go to Kubernetes commands because I forget that stuff, especially the gnarly ones. Like, how do you freaking go through your logs in Kubernetes? Or how do you log into your pod? Into your container in your pod? Or be like, yeah...ROB: I'm with you, man. I have a folder with documents, and it's for, like, Kubernetes, Docker this, and I'm like, get lost in those. And then it's like searching through those commands. So we're applying this Enterprise search and conversational search to Kubernetes and Observability. And it ties into AI Ops. So I'm in awe in how powerful large language models are and applied in the right case, I'm going to write a blog. It's on my to-do. I kind of have a draft format where take it from a different angle, right.ADRIANA: Yeah, yeah.ROB: How these AI tools can help the world, right. I see too much boom and doom, kind of, hey, they're going to break this, break that. It's going to take cover, control. Yes, everything, right? We have the biggest case of nuclear power. It's for good, it's for bad. It's our human choice to use it for the good. Right. And I'm always optimist.So I love it because it can apply to so many...We just combined the few elements, AI search and Observability and Kubernetes and boom. That's something we're working on.So that goes back to engineering and working cool stuff. So that's kind of what I really enjoy.ADRIANA: That is super cool. Yeah, it's funny because I think AI has definitely become a hot topic because it's come up more than once in this podcast. I think my first dabbling into AI was, like, using DALL-E for generating images for my presentations. That was kind of my first one where I'm like, oh, my God, this is the coolest thing. I can tell it to generate pictures of llamas doing funny things. What? Or my favorite, I have this love for capybaras now because Instagram one day decided to serve me pictures and videos of capybaras.And I'm like, oh, my God, this is such a glorious animal, you know, DALL-E has generated me a bunch of images of these things for my presentations as well. So I'm like, "Shit, that is some really rad stuff."And then further leveraging ChatGPT for even certain things, where you find yourself in a position where I need to reword this thing. My brain is fried. "ChatGPT, just take the sentence that I wrote and make it a little bit shorter," because I don't have the brain power to try to think of five different ways of saying this word and conveying this thing, right?ROB: So you're touching on something pretty cool, right? So it takes you to the next level. And some people say it actually does it for you. It doesn't yeah, it's going to be mind-blowing. It doesn't, because I can smell, like when a marketing person talks about technology thing, and it kind of doesn't make sense. And then when a techie will use the same kind of and they will just rephrase it. There's a difference.So it's such a helpful...I love it. It's been changing. So we applied it kind of all over the place. Again, combining AI, Observability, DevOps like...crazy.ADRIANA: It's going to be mind-blowing. And I think people forget that it's not like AI, as you said, AI is not going to do all the work for you. You still need the human touch to guide it in the direction, and then you still have to vet it because sometimes AI spits out some dumb-ass shit and you're like, "No, I do not want this." And then you just rephrase the question.At first when I heard the term "prompt engineer", I'm like, "Ha ha. That's so hokey."But we've been prompt engineers for a while now, if you think about it, in software, because that is essentially what we do when we do a Google search, especially when we're trying to solve a gnarly-ass problem and you enter a particular search term, and then you're like refining, refining, refining, until you're like, oh, you know what? That's not even the right question that I have to ask, but now I've got enough information that I know the right question to ask, and that's essentially what a prompt engineer does. It's just now the floodgates have opened in terms of what it provides you right. It's more than just those Google search results. It's more contextual information. ROB: So, you know, I agree with you. 100%. So I didn't know what prompt engineering was. That I was doing prompt engineering, right? Before it was...because I was doing what you said. It was kind of like the engineering brains, like, okay, I'm going to do it this way. I want to ask it that way. Oh, it's pretty cool. And then you start learning from it and then yeah, you were engineering a prompt, right? As a CEO, write me an email on this promotion.ADRIANA: Make it sound more beautiful. Another thing that I want to ask is, we both came about in technology before there was such a thing as Cloud, Kubernetes...We are children of the monolithic era of Java enterprise servers, which are no longer I don't know if I miss it or if I'm glad that that stuff's gone. What was your foray into Kubernetes? What led you in that path?ROB: So I was doing consulting in Montreal, this is...whenever Docker 1.1 came out and was lucky enough that the company was kind of looking and really trying to find solutions around Docker. And we use Docker Compose and Docker Compose is kind of limiting solution.And from there, just bringing the Docker world, we kind of started working with it. We had a few implementation of Docker Compose for clients, and then Kubernetes came, right? Early adapters...and kind of jumped on that because there was a limitation of controlling and deploying Dockers without a kind of orchestration platform. So we kind of started building for Google and Kubernetes and creating our own kind of platform with the CLI on how to deploy, ordering...So we were kind of early kind of working on it and the tools that we have right now weren't there. We kind of build them ourselves. So that's how we jumped on on it. So it was through a client and then just thinking that it's really cool how you can kind of abstract to OS level virtualize, a little kind of component.It was just kind of groundbreaking even though Linux had it before putting it in kind of element where, hey, we're using Eclipse. And then instead of deploying MySQL on my Windows box at the time we deployed it in Kubernetes...sorry...in Docker, which you can kind of start and turn on and off.It wasn't some kind of heavy windows or Mac installation.That kind of it was just bring it up, the Docker is there, and connect to it. And I was like, man, that's pretty cool, right? And I'm like, man, I don't know. As an engineer, it was kind of groundbreaking tech porn.ADRIANA: Yeah, I totally agree.ROB: I'm like, oh, my God, what can you do? And then not a lot of people were working on it, but we had some solutions, and then Kubernetes was the next kind of level.ADRIANA: Yeah.ROB: Kubernetes is complex, right. It's not easy. I wouldn't say for everybody to use it. There's a good case for it, but those benefits that it brought were pretty cool in terms of kind of working with containers and providing the networking and deploying. So kind of building around that. That's kind of our first foray to it. And it just continues until now.ADRIANA: I think that's such a really good point on the containerization is the gateway drug, right to Kubernetes. I mean, it really is. Docker in itself was awesome. And then you're like, oh, shit. Now I've got to manage these Docker containers in tandem and figure out all this stuff, the networking and stuff between them. And then Docker Compose kind of helps you with that, and you're like, okay, that's better. And then you realize I need a little more, umph.Oh, Kubernetes is like, the next natural evolution of it, where you're like, oh, my God, this makes things so much easier. But then at the same time, it's like, my life is hell. It's like, you can't win, right? It solves a problem, but then it brings on additional complexity because it is such a complex tool. But so cool.ROB: Yeah, it I keep following kind of...some questions. Once use Kubernetes, and people are against it and big projects, small projects, I have a simple answer. The community of tools is so big right now, you got to use it because everybody's kind of working towards one goal, and that's the beauty of it, right? Yes. It's complex. Yes. It's hard. Yes. You got to have that's what we try to make it easier. Yes. You got to remember that managed Kubernetes is a little bit easier, but dealing with it overall, it brings complexities. But having every single tool, like Cloud Native tool, you go into a landscape, every single tool is deployable on Kubernetes, right?So having that power and building from infrastructure-as-code and kind of Helm Chart and combining it all together, the power is there. That's kind of what I think is the biggest benefit. So, yeah, use it and then use it smartly. If somebody asks you when to use it or if it's good or bad, man, that's the wrong question. You find a problem, and there's solutions for it.And if you want to build a WordPress site, build it on Wordpress.org or something, right? Or if you want to deploy WordPress and Kubernetes, deploy in Kubernetes. What is your need? What is your problem?ADRIANA: I totally agree. And it's funny because I was having a similar discussion with folks today where I was chatting about Kubernetes and Nomad and how a lot of people talk about it in terms of a versus thing. But it's like, what is your use, case? When I worked at Tucows, it was a Nomad shop.And it made sense because they had their own data centers, which meant that when they tried to start up their own Kubernetes clusters in their own data center, that's like you are creating your clusters from scratch, which is a horrible, horrible experience. Versus if they were using Public Cloud and have access to managed Kubernetes, maybe that would have changed the conversation.But at the time, using data centers well, between running Nomad in a data center versus running Kubernetes in a data center, it's a lot easier to manage a Nomad cluster compared to a Kubernetes cluster. But then also, I guess some organizations might not need the additional complexity that you get with Kubernetes, and so they might choose Nomad or whatever other product because there's like, for example, VMware Tanzu, right?They're a competitor as well in the space. I've not played with it, I've just heard of them, and that is the extent of my knowledge. But it's interesting to know that there are other competitors in the space that solve the problem, but in a different manner. And maybe that suits your use case better.ROB: Yeah. So, when we were working, like, a few years ago, I felt it was nightmare to have Kubernetes on premises or data center, to your point, right?No matter something, the tools were not baked in. Now it's easier. But that one leads me to a question for you. What do you think of...I read some articles that were kind of, I guess, headline grabbers. Cloud is dead. People are going back on premises or data centers.How do you feel where the world is going to go? Like, having a crystal ball...Cloud versus maybe people going back to their own data centers or hybrid. Any ideas there?ADRIANA: Yeah, I think it's going to be a hybrid thing because here's my take on Cloud. I think Cloud abstracts a lot of the complexity that you would have for managing your own data center. And I think to a certain extent you can even manage the complexity of running your own data center through tools like OpenStack. And I think Azure has a thing called Azure Stack, and I'm sure the other Cloud providers have their own thing as well.So you're basically having the same nice little infrastructure-as-code convenience in your data center rather than hosting in Public Cloud. Now. I think a lot of people treated like there was this mega rush to public cloud, I think because A, it was easy, and B, there was a lot of hype.And then people forgot to look at the cost, where they're like, oh, this stuff is limitless. No, until you get your first cloud bill and you're like, "Shit, that was a massive cloud build."Did I actually need all that stuff? But in terms of leaving it to somebody else to manage your infrastructure, awesome. But you have to be super mindful of your costs. Whereas when you're running your own data center, you are so mindful of your costs because you are keeping an eye on that budget like a hawk. Right? It's like, no, I do not have extra rack. Like, I ran into an issue when I worked at Bank of Montreal where we were setting up...we had to buy new physical server.There was no rack space. They had to buy a rack. And because there was no rack space, they had to lay in the electrical work to be able to rack up that server. There was all this stuff that you take for granted when you're, when you're running in Public Cloud.ROB: Yeah, I think it's going to be both. I'm just a proponent that it's really hard for data centers, specifically the smaller ones replicate the security, right? How do you do that, right? So you have these billion dollar clouds and their day in, day out as they go to an office and they think how to make it better, how to make it better. And over there where yes, now there's great tools from a lot of clouds to have infrastructure kind of as code infrastructures, cloud foundations for your data center. But that investment, continuous investment into securing it, that's what worries me, right?Maybe like to your point, I'm hearing horror stories with managed services and cloud and cloud bills that it might be more kind of cost efficient to have that data center. Right. Because the cloud costs are so huge. Right. I think we're kind of still in the early stages, but I think it's going to be hybrid. I just don't know how cool will solve that issue of having a secure data center. So is it going to be your own data center stack?So maybe know Bank of Montreal or the big banks or Canadian Tire might have their own because they have money for it. Right. And then you might have data centers that are kind of from the old age where they host stuff for you and you just have your rack in there. We might solve the cost savings issue, but if we don't, we're going to see some bigger blowback. But I just don't see yet how other companies can replicate that heavy investment those big three are doing into that security or whatever security or the future tools or that's kind of where the word is going to be. So I'm going to see where it's gonna go.ADRIANA: Yeah, it'll be interesting and in the same way way that you're kind of keeping an eye on the whole data center situation. On prem or Cloud. I think we'll see a similar movement with the monolith versus the microservice, because, again, a lot of organizations rushed into the microservices model thinking, this is going to solve my problems. And then now they're rethinking it, which is rather interesting, which I'm not surprised by, because, again, it's like a lot of hype and a lot of people just did very knee-jerk reactions, rather than, is this actually going to do the thing that it's supposed to do for me?ROB: I think it is going to go somewhere in between. As long as you work towards the scalability and elastic nature of the cloud, build it for that, right? So microservices are good for that if built well because you can isolate the problem, right?If you can have a monolith, make sure you can do the same thing. Make sure you can scale the biggest thing of monolith. Once it couldn't scale and you had your 30 different features in one set, and then what, right. So there's room for both. And it's an architectural pattern they want to use.I agree. But it's the same answer. The Kubernetes answers. It's the same kind of answers. When I see kind of, hey, I go to Reddit and I'm Kubernetes and I'm here and I'm there, it's the same thing. Oh, I would never use monolith. I'm like, man, that's not the right answer. Be more a little bit critical of what you're trying to say, why you fail with your problem. It's not like brass stroke for everything is the same. So room for both.Got a question for you. Can I ask? Okay. Observability...what do you think of same thing with the cost because you're in the space and I think we had a conversation on it before, manage versus kind of in your stock because just an example, DataDogs and all that stuff, the same thing. You start slowly and then there's a boom, a bill, right? Is that bill justifiable millions of dollars? Where do you stand in that Observability world and what do you think about open source or in your kind of open source entire Kubernetes or kind of powered by open source versus kind of the fully managed solutions and the benefits kind of of that. Where do you stand with that?ADRIANA: Yeah. So I'm going to put on my not a "I work for an Observability company hat," but my "I was in the position of managing an Observability team hat," and from that perspective...so when I worked at Tucows, I came in to manage two teams, a platform team and an Observability team. And the Observability team at the time, their function was basically managing tools and not focusing on practices. But we were also using a SaaS vendor. So internally managed tools plus SaaS vendor. I'm like, you know what, you've already got the contract with the SaaS vendor. Let's use that as the standard. Let's ditch the internal tools so then we can focus on practices and focus then on making sure that people are doing Observability properly and making sure that we standardize on the OpenTelemetry.Because this was like the early days of OpenTelemetry, so traces weren't even general availability. Now we're at the point where traces are general availability, metrics are general availability and I think logs are stable, but depending on the language, it's like the specification is stable, but it's on a per language basis, like where things are. But long story short, OpenTelemetry has evolved a lot and for me it was more important coming into that team making sure that the organization was doing Observability properly rather than focusing on maintaining tools. Because if you're so focused on maintaining tools, then what's to say that you're actually doing Observability properly? So we wanted to set out a set of best practices across the org.Now, we did run into cost overruns with the vendor that we were using, but the nice thing about using OpenTelemetry is it gave us this opportunity to...because my focus was, let's make sure that the organization instruments everything in OpenTelemetry. And they were not. They were using vendor SDKs at the time. But my goal was let's inform people on making sure that they adopt OpenTelemetry so that if you're stuck with a vendor that way you're not stuck with a vendor that's going to cost you a gajillion dollars.Right now you have that flexibility of going to another SaaS vendor or...you know what, now you have the flexibility too. If you want to go the self-hosted model you have that kind of flexibility. But yeah, I feel like when you're evaluating vendor, you have to know what you're getting in bed with. Because as soon as with that particular vendor, we started moving away from their SDKs and started using OpenTelemetry, the cost shot up because they supported OpenTelemetry, but they treated the OpenTelemetry stuff as like extra I don't know, extra nodes or whatever, extra containers or some I forget what it was, but our costs shot up. It was shockingly horrifyingly expensive as a result.So I think you need to understand the cost model up front. Unfortunately a lot of vendors have very complex costing models which then that makes it a little bit tricky. Yeah.ROB: So when you said that if you design it properly, do you think you can very easily exchange the tools because your best practices are kind of build on OpenTelemetry and then you can kind of go from tool to tool? Is that kind of what you mean by best practices?ADRIANA: Yeah, so best practices means...because the idea of Observability is your system is emitting enough information so that even without knowing the inner workings of the system, you have enough information so you can tell what's happening, right? So yeah, you can use OpenTelemetry but if your system is not emitting the right stuff then so what, right?And it's a combination of emitting the right stuff and also making sure that the vendor is representing the information. So then when you instrument using OpenTelemetry the thing that differentiates the vendors is how they render that information. Is this going to be useful to you? So it's a combination of making sure that the code is instrumented properly and also is this thing showing up in a way that's useful to you so that you can troubleshoot. Right, so that I think becomes the trick.ROB: Yeah, that's good, right. It's kind of like what we're concentrating with kind of our stack. But the journey is not understood, right? And I feel some vendors are overselling the promise because the tool will not solve everything and you can just get into a really bad practice of paying a lot because you're going to be searching for what to collect and just scraping everything possible. So that best practice we're talking about and then emitting the data, collecting data. That's a very important piece.So back to the other question. So we have the practices and kind of OpenTelemetry and kind of instrumenting the code. Where do you find then after that's done, the SaaS model vendors and I don't want to pick on DataDog, there's a few others of them that are there. Where do you feel they fit into that once you have that set up the internal platform versus external SaaS model.ADRIANA: In terms of what specifically?ROB: For Observability. So comparing having Prometheus stacking your Kubernetes versus maybe connecting to again Logs.io, just say, right? Because they're kind of API based and kind of instrumenting kind of thing. Where do you think do you have an approach or preference towards one or you think it depends on the situational and company?ADRIANA: Yeah, I think at the end of the day it just depends on your situation. When I started my Observability journey, my dream was to have a tool that took care of all the things. So in my ideal world you could do away with Prometheus because you can emit those Prometheus style metrics and then just ingest them into whatever system and you'll have a place that displays your metrics, your logs and your traces and they're all correlated nicely.ROB: Right.ADRIANA: I don't think that any one vendor does that well right now it's interesting too, like for example in OpenTelemetry there's a way right now to correlate your traces and your logs which is currently being implemented. There's a way to correlate your traces to your metrics. It's called a trace exemplar. But when you look under the covers...so a lot of people talk about trace exemplars. You look under the covers. It's not been implemented for a lot of languages. I think the only one that's actually been implemented for is Java. So then you'll see actually a lot of vendors that will do that correlation in the tool itself and not use OpenTelemetry for it, which is quite interesting. So there's still some work to be done. It'll be interesting to see where things go.ROB: That's an interesting problem that I feel we always face because they're so wide to kind of adapt to so many different languages and tools and stuff and open it up and making sure can one company be doing everything well? It goes back to kind of can Apple do everything well? Can Microsoft do everything well? At what point can you invest in everything, right?So that's going to be interesting to see when I was talking to somebody at a conference, what's going to happen eventually is people are going to be really buying out each other, right? We're going to reach that level where they're going to be eating up and then, hey, these guys are doing good. This level Observability, combine it together and then see if that works.I spoke to you about it as well, kind of because where you are so that's going to happen. That's good and bad. Because that will kind of go to your point where maybe somebody's going to be able to create that kind of one tool by waiting to see if there's going to be enough appetite and investment to make those different parts of the tool well structured. So it's pretty cool. Pretty cool.This whole Observability is just so crazy, so vast. You can spend just like and you can spend a world and all your time reading about it and you still can kind of tackle the fraction of it, right?ADRIANA: Oh, yeah, absolutely. I know. I do this for a living, and I'm like, I've barely scratched the surface. Well, cool. We are just coming up on time. So, for parting words, do you have any awesome advice that you want to share with our lovely audience?ROB: Go slow, talk to experts. If you do things, try to do them right the first time, but don't be afraid to fail. And iterate, right? So it's kind of challenging aspect there, but yeah, maybe for people that are starting out, touch technology, it's here with us. For AI, embrace it, don't hate it. It's here with us. There's ways of things, figuring it out. As long as we have a positive outlook for what we want to do, we're humans are very smart, we're going to solve it. So that's kind of the approach I take too. All these different things that are coming out, maybe because we're techies, we enjoy it more because we see the potential of it and I see huge potential and just where the world's going in a very good way, very positive way.ADRIANA: Totally. That's awesome. Those are great words of wisdom. Well, thanks so much, Rob, for geeking out with me today, y'all. Don't forget to subscribe. And be sure to check the show notes for additional resources and to connect with us and our guests on social media. Until next time.ROB: Peace out, and geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:Luiz Aoqui (he/him) is a senior software engineer at HashiCorp working on Nomad.Find our guest on:X (Twitter)MastodonLinkedInFind us on:All of our social channels are on bento.me/geekingoutAdriana’s X (Twitter)Adriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:Go (Golang)HashiCorpHCLNomadJava JARAmazon EKSGoogle GKS (GKE)Microsoft AKSNomad Agents and ServersVault (HashiCorp)Terraform (HashiCorp)PodmanManaged KubernetesKubernets OperatorNomad Operator patternnomad-ops projectKoyebkreconcilerKubernetes CRDNomad 1.6Nomad node poolsKubernetes node taints and tolerationsKubernetes node poolNomad datacenterNomad EnterpriseNomad scheduling algorithms (spread, binpack)RFCNomad on Kubernetes Tutorial - Kelsey HightowerDocker-in-Docker (DinD)K0sBlog post: Running K0s on NomadetcdKubernetes controllersKubeletk0s on ARM machinesHashiqubeVagrant (HashiCorp)Consul (HashiCorp)Jaeger (tracing tool)k0s - cgroupnscgroupns flag - Docker runLuiz Aoqui on On-Call Me MaybeLightstepHoneycombZipkinNomad PreemptionDProfGo OTel auto-instrumentationeBPFTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And geeking out with me today is Luiz Aoqui. Welcome, Luiz.LUIZ: Hi, Adriana. Nice to be here. Thank you for having me.ADRIANA: Thanks for coming. Where are you calling in from today?LUIZ: I'm calling from Toronto as well.ADRIANA: Woohoo.LUIZ: Representing.ADRIANA: Fellow Brazilian in Toronto. Awesome.LUIZ: lots of us here.ADRIANA: Yeah, there are lots of us here. So, we're going to start off with some lightning round questions before we get into the meaty bits. So let's get ready. Be prepared. I swear it's painless. Okay, question number: one are you left-handed or right-handed?LUIZ: Right-handed.ADRIANA: All right, question number two: iPhone or Android?LUIZ: Android.ADRIANA: Number three Mac, Linux or Windows? What's your preference?LUIZ: Going to say Mac for now.ADRIANA: For now. Awesome. Favorite programming language?LUIZ: Go. Pretty big favorite of mine.ADRIANA: Awesome. Dev or Ops?LUIZ: I like both, but I have to say dev.ADRIANA: Cool. No, wrong answers. JSON or YAML?LUIZ: I think I'll pick YAML. Just more friendly.ADRIANA: Fair enough.LUIZ: I hate the lack of dangling commas on JSON.ADRIANA: Yeah, I agree. I hate that too. And then finally, do you prefer to consume content through video or text?LUIZ: Text. Yeah, I get very distracted watching videos.ADRIANA: Yes. Same. Awesome. You survived the lightning round. By the way, I should mention we had someone on here who when I asked her JSON or YAML, she said HCL.LUIZ: Yes. That's why I had it in my back of my mind, like, oh, I only have two options.ADRIANA: So I thought that was pretty funny when she mentioned that. And then I thought of you, because for those who don't know, Luiz works at HashiCorp, he is a Nomad developer, so HCL fits well into the type of work that you do. So I wanted to start off with you being a Nomad developer. Tell us a little bit like, for folks who aren't familiar with Nomad, tell us a little bit about what Nomad is.LUIZ: Sure, yeah. So Nomad is a workload orchestrator, and I know that doesn't mean a lot to many people. So the goal of an orchestrator is to basically get your assets...So, like, your developer team build things like Docker images, binaries, Java JAR files, whatever. They produce some kind of artifact out of source code. And then you have your infrastructure team that is responsible for running your infrastructure, building servers, configuring machines, all of that. And then in the far end, you have your users that are trying to access your product, trying to access your application. And then the orchestrator sort of sits between your development team and your infrastructure team to make sure that whatever artifacts gets produced, it's running on those machines. So it helps finding where to run things like figuring out what's the best server to run this application, or doing things like upgrading and deployments and all the sort of lifecycle management of your application. That's the job of the orchestrator. That's what an orchestrator do.ADRIANA: Awesome. Yeah, and I think that's such a great way of explaining what an orchestrator does, because for folks who are familiar with Kubernetes, I mean, Kubernetes is an orchestrator as well, specifically for containers, whereas Nomad gives you that breadth of, pretty much orchestrate anything, more or less. But it is very easy to kind of forget all of the gnarly things that happen behind the scenes in these orchestrators, like all the hard work that they're doing in order for them to operate seamlessly.Now, I've played around with both Kubernetes and Nomad, and I have to say one of the things starting on the Kubernetes side and then moving to Nomad, moving to Nomad was actually a lot easier because you kind of dealt with the complexity of Kubernetes moving down to Nomad. You're like, "Oh, it's like the simplified...everything's simpler." And it runs in a single binary. It can run in a single binary on your machine and you can get started easily, whereas Kubernetes is more of a beast. I mean, yes, you can have really complex setups with Nomad, of course, and that's probably how you have it in production. But as far as I think the barrier to entry when it comes to Nomad is very low, which I think can be very appealing.LUIZ: Yeah. Complexity is an interesting discussion because complexity sort of means different things for different people, I think, when thinking specifically on this Nomad versus Kubernetes discussion, I think there are a few things to consider when you think about complexity of adoption. Let's say when you start, if you starting from a managed service for Kubernetes, like EKS, GKS, AKS, that's like one click, and then you have a cluster. So like, oh, there's no complexity there. And that's how most people nowadays consume. Kubernetes is through a managed service. So that almost basically removes the barrier of entry for those that are able to use a managed service, of course.But then it gets the complexity of understanding how to use those systems. Like, okay, it gets provisioned for me. There's some cloud magic happening behind the scenes. I don't have to deal with that. But now you have to run the system. Now you need to think of how do you map your team's workflow to that new tool? So there's all these different concepts in Kubernetes that I think is part of the complexity. There's all these different tools that you can use. Having a broad ecosystem is great, but it can also lead to some confusion about when do I adopt to do this or when do I do that, how do you bring all together to eventually reach your end goal?Which is like, I want my users to be able to access my product right. When I think complexity, I think more on that sort of day-to-day operations, like understanding what's happening with the system. And I think that's when sort of Nomad becomes simpler just because it has a smaller surface area for people to interact. You write your job, you run your job, and there you go.But like stepping back a little, the complexity of Nomad comes in on the deployment part because we don't have a managed service of sorts. Like, okay, now you need to understand what are Nomad agents, what are Nomad servers. Now you need to manage their state, now you need to manage upgrades. And this can get complex in that sense. So I think that discussion of complexity, I find it very interesting just because of this duality of like, okay, what am I trying to do? Am I actually running the cluster? Am I actually just using it? And somebody is provisioning...nowadays we call the platform teams. Is there a platform team running a cluster for me? And so yes, it's no much simpler than Kubernetes, I guess, depending on what you like, depending on where you're kind of like depending on what complexity you're talking about.ADRIANA: Yeah, that's so true. That's a really good point. I just want to go back to a point that you made earlier about the fact that there's all these Kubernetes managed services, but there's Nomad managed service, which is kind of interesting, because if you look at various Hashi offerings, there are managed services for a bunch of stuff. So why is it that there is no managed Nomad right now, even offered through Hashi's own cloud?LUIZ: Yeah, it's something that definitely part of the plan, is something we think about. But there's quite a few challenges for Nomad, specifically compared to other HashiCorp tools, is that we have a big competition with Kubernetes. So if you have a managed Vault, that Vault, this is basically the only tool you have.You have a managed Terraform, and there are other tools for infrastructure-as-code, but like, Terraform is one of the big ones. For Nomad, this competition is much stronger, in that sense. Where you're...so, that's mostly my personal opinion.But thinking of a managed Nomad service versus a managed Kubernetes service, it gets into that point of how much value that actually becomes. Because part of the benefits of Nomad is the flexibility and the flexibility of running different environments, flexibility of running different workloads. But those workloads, you sort of need to have control over infrastructures. Like, I want to use Podman, so I need machines that have Podman installed.ADRIANA: Ahh, okay, got it.LUIZ: Going back to that discussion of managed versus self-hosted, it's a spectrum, right? Like if you go to managed, it's simpler to operate, but it's probably more expensive. And you also lose flexibility, whereas in self hosted it's harder to manage, but you get full flexibility and probably cheaper. So there's that aspect that sort of like, we need to figure out.There's also something to consider about costs and things like that. Because if you have a managed Kubernetes service on AWS, AWS is sort of taking the heat of like, they can sort of discount the compute because they run the compute.But now if you have an external managed Nomad, you need to account for the price of that service plus whatever infrastructure you're using. And so it kind of becomes like a pricier solution. And so figuring out the business part of that can also be a little challenging as well. So there's all of these little things to consider, but I have to see what happens in the future.ADRIANA: It's a really great explanation. Thanks for clarifying. Another thing that I wanted to ask you about is, I think one of the big things that Kubernetes has that you don't see, I don't think a direct translation for In Nomad is that you've got the whole Kubernetes operator concept, whereas as far as I know, there isn't that concept quite in Nomad. I've seen a few blog posts where people try to replicate it to a certain extent, but it's not quite the same thing. Can you comment a little bit about that? LUIZ: So that's a very sort of common question to see. And I think there's this idea that you need to think about that operators is just a pattern that you can implement in pretty much anything. So the idea of listening for events and responding to them and that's something you can do with Nomad. And actually a few projects that I've seen do that. I was googling for the name. There's a project called nomad-ops that implements the operator pattern in Nomad using some of the functionalities we build.There's also a company called Koyeb. They are a platform-as-a-service company that use the operator pattern. And they have a library called kreconciler, I think it's called, that helps you build this sort of operator paradigm functionalities. But it's also important to think that on Kubernetes, it's not just operators that are the main thing. When you talk about operators, you're always associated with a CRD because that's the data. So you have operators being the logic, CRDs being the data. And that sort of helps guide your end goal sort of based on those two concepts.So in Nomad, you can do the operator. We're building things that can help you do that sort of things. One of the challenges, like how do I access the Nomad API from my task? And we're building like, now you have like a socket that you can use to talk directly with API. We are building workload identity, so you don't have to worry about ACL tokens or anything like that. It's like we're building things that help people create the sort of operator paradise in Nomad.But the CRD sort of becomes a bit of a challenge because we don't have that concept of as extensible as Kubernetes has. But you can sort of have that Nomad variables and things like that. You can kind of get around those challenges. But I think there's no another point of CRDs is that they're sort of standardized, right? Let's say the Prometheus operator expects to have this or generates specific CRDs. And then a Grafana operator can sort of rely on those CRDs to do stuff automatically. So this type of standardization, we don't quite have that yet.ADRIANA: Right. Interesting. That's really cool. I'll definitely be sure to put those two projects that you mentioned in the show notes for folks to refer back to those. Another thing that I wanted to touch upon, because I believe you and I talked about the fact that there was a new version of Nomad that just came out. So what version just came out? And what are the exciting things that we can expect to see in Nomad?LUIZ: So Nomad 1.6 just came out and the main feature is called node pools. And something that I worked on, so apologies for any bugs. The idea of node pools is that it allows you to create...sort of like, segment your clients into groups, into pools.So a bit of Nomad background very quickly. So you have two types of machines in Nomad cluster. You have a servers. It's like your control plane. They do the scheduling, they store state, they sort of do all the global view things of your cluster. And then you have clients, they're like talking to the servers to get information about what that specific client needs to run.So the client is sort of the data plane is the component that actually run things, so actually runs your Docker containers, your JAR files, whatever. And so one of the challenges we've had in the past with Nomad is that it's very hard for you to sort of associate a group of, let's say I have a group of clients that I want to run my backend services.And you can do something like create a constraint that says, okay, my back-end service only runs on machines that have this specific metadata. But doing the opposite is kind of hard because constraints are like, you need to tell a job what the constraint is, but in order to sort of prevent others from accessing those same machines, you need to create like a negative rule. So you have to say, okay, this machine, this job runs on these machines, but every other job is forbidden from accessing those machines. So you sort of have that dual constraint type of thing. So it's very hard to manage that to get a consistent scheduling outcome because of like, if you forget a constraint rule, now your job is running somewhere that it wasn't supposed to be.So with node pools, you can put a new configuration on each client saying which pool it belongs to, and then on your job you'll say, okay, this job runs in this pool. Now only jobs in that pool will access those clients, and those jobs will only run in those clients. So you kind of create a sort of segmented part of your infrastructure that is reserved for specific jobs.ADRIANA: I want to say what you were describing without the node pool is kind of reminding me of Kubernetes node taints, where you can say what can run where.LUIZ: The idea is to have a very simple solution. So it's like this node is in this pool, this job runs in this pool, and that's all you have to do. So there's not a whole lot of configuration that you have to do. And in addition to having this sort of segmented view of infrastructure, node pools can also hold configuration. So they are a first-class concept. In Nomad, for example, one of the workarounds people used to do to get around this constraint problem is to use data centers.So in Nomad, the idea of a data center is just like a collection of nodes. And if you have an availability zone, that sort of becomes your data center. And people would kind of hack around the problems using data centers, so they have like a data center for apps, which doesn't quite match the intention behind it.But the problem is that data center is sort of just like a metadata, so you cannot have specific configurations per data center, things like that.ADRIANA: Okay.LUIZ: But with node pools, you can attach like you can put a description on your node pool, you can put metadata in the pool to sort of create more information about what this pool is used for. And then in Nomad Enterprise you can actually have different scheduler configuration per node pool. So for example, you can have a pool that uses the spread algorithm and other pool that uses bin packing so you can adjust how the scheduling is done per node pool. So there's a bit of extra customization that you can do per pool and that could be very helpful in several cases.ADRIANA: That's awesome. It's interesting because there's this recognition of, like, oh, people were kind of trying to hack together the concept of a node pool by using these data centers. So it sounds like there's this recognition of oh, users are trying to do this. Why don't we formalize it and turn it into a proper solution? What I like that you had said earlier also, which I think it feels like it is a general philosophy with Nomad is basically going for the overall simpler solution. Like, don't try to overcomplicate. Just go with the base thing. That works pretty well and we don't have to drive ourselves mad.LUIZ: Mmhmm, yeah, exactly. Yeah. And that was a big thing during the development phase because a bit of background on how we develop features at HashiCorp, we start with an RFC. So whenever we want to implement a feature, we write down the description of the feature and sort of send to the we first sent to our immediate team, then to the whole company for feedback. And during that process I had gigantic ideas like, oh, maybe node pools should be dynamic and then you can dynamically add and remove nodes from the pools. But that sort of adds so much complexity with questionable value.So that's like part of the feedback I got from the team was like, let's start simple, let's start solve the problem at hand and then we can expand if the need arise. But yeah, this idea of simplicity, trying to make things easy to use from day zero, it's very important to us.ADRIANA: Awesome. That's very cool. I want to switch gears now because there's, like, two things that I'm hoping that we'll have time to talk about still because there's so much cool stuff to talk about, but I want to switch gears quickly to a collaboration that you and I did, which was really fun. It came out of just me having a wild idea that came out of nowhere, where basically I thought, "Hey, wouldn't it be cool to try to run Kubernetes on Nomad?" Because there's Kelsey Hightower's well known GitHub repo where he's running Nomad on Kubernetes.So I posed the question, "What if you can run Kubernetes on Nomad?"And I thought, "Maybe let's not try to go too crazy here." And so my idea was, like, I want to find a lightweight Kubernetes distribution that we can run on Nomad, something that's hopefully already containerized, because trying to run Docke-in-Docker is kind of a nightmare if you try to do it yourself. And so there's this distribution of Kubernetes called k0s that comes in a Dockerized version, which seemed relatively straightforward to deploy on Nomad.And so I reached out to you when I came up with this idea, and then you helped me through a bunch of the troubleshooting, so I just wanted to talk to you, have you share your experience around this collaboration. Yeah, just thoughts.LUIZ: Yeah, it was pretty fun. Like a lot of learnings, I think, in terms of just understanding how things work, because k0s is pretty cool project in the sense of like, oh, you just run that image and you get a container with everything sort of there for you. But I found it a nice learning lesson of debugging of when things don't work. So normally you would expect things just "docker run" and it works. But what happens when it doesn't work and having to debug and going through logs and sort of combing through those log lines. I found it a bit challenging because having everything in an image, it's easier to start things. But then when you need to debug and you have your etcd logs at the same time as your controller logs, at the same time as the Kubelet logs, it all sort of juggles together and it was very hard for us to sort of comb through that and understand what's actually failing.ADRINANA: That's so true. That's so true.LUIZ: And then you have retries, right? So you see an error message and then it retries and then there's an error message again. But is it the thing that is retrying that is the problem or is it trying to call something else that failed before and then the log just sort of disappeared from the history? Just because that part, I found it very interesting.ADRINANA: Yeah, I totally agree. And it was funny because I did the classic rookie mistake of like, well, of course this thing works in Docker. Let me just try to deploy it in Nomad. And then I realized I was getting all these error messages where the Kubelet was not starting, which you kind of need that for Kubernetes. So it looked like it started up in the Nomad job, right? The Nomad job deployed successfully, but the actual thing inside the job was not running correctly.So I neglected to try to run the k0s locally on Docker and then discovered a bunch of stuff where initially we were running into all sorts of issues, too, because running on M1 Macs, everything is special. I love having an M1 Mac, but, my God, there's all these little annoying considerations. So that made it extra complicated. But then once we got it running standalone in Docker on the M1, then we were able to port it over.And it was interesting because for me, I always like to try everything in Hashiqube, which is like this full-fledged Nomad environment where you have, like, Nomad, Vault, Consul all running together. But it's provisioned using Vagrant. And on the M1 Mac...normally you'd provision using a VM in Vagrant...but on the M1 Mac, Vagrant does not play so well. So it's Docker as the provisioner. And so you're basically running Hashiqube. So you're running Nomad in Docker. And then you're deploying Kubernetes as a Docker image and then deploying our test app, which was Jaeger. Was Docker in Docker in Docker. It was like, I don't know, three or four layers of Docker.And then you took the more pragmatic approach of, like, let me just run this using the Docker binary. Sorry, the Nomad binary. Much easier.LUIZ: Yeah. That applies like several scenarios when a GitHub issue shows up, people describe their entire cluster and environment and give us a ginormous job file. And usually my first step is, okay, I need to reduce this, you need to boil down toADRIANA: It's so true.LUIZ: What's the actual problem? So, okay, let's try to remove, let's say, half of the job that I don't care about.ADRIANA: Yeah, I think that is a very sound approach.LUIZ: Let's just run a dev agent, see if that sort of reproduces the issue. Normally, my first step is to reduce as much as I can and then start adding things. So, yes. I don't know. Dev agent for me, Nomad is like my direct go-to anytime I need something. Nomad, "nomad agent -dev" and start from there. Yeah.ADRIANA: Reduce the noise as much as possible and then start building back up until you figure it out. This is the wall I actually hit. So, yes, lesson learned to you all. I should know better. I've been doing this long enough that I have found myself in situations where I want to do everything all at once. And then I'm like, strip, strip, strip, strip, strip all the things until you get to the actual problem. But this was a fun little collaboration.And then there was one component. What was it? The C groups namespace where there's a Docker configuration that Nomad did not support. And so this is where it helps knowing somebody that works on Nomad because Louise was able to make, like, a little fix to accommodate. It's not part of the Nomad product. So you will not find this as part of standard Nomad. This was just so that we could see if we could get this running with this configuration.LUIZ: That part is very...on the surface, it's like, oh, it's like configuration value that it passed to "docker run" on the Docker CLI, that's just a flag that is set. But what it actually does, it's much deeper onto the environment that you're the I forgot the exact flag.I think it's cgroupns. And then you need to put host.ADRIANA: Yeah, I think that's the right one.LUIZ: But the tricky data is that Docker and cgroups, they do, like, weird stuff that Nomad sort of needs to work around that to make some of the Nomad things work. So, for example, resource isolation. Nomad uses cgroups to sort of enforce that. No matter what task you're running, no matter if it's a Docker container, a binary, JAR file, we use the cgroups because that's the common layer. But the way that Docker does things, it kind of hides that from you. So you as a developer, sort of needs to work around all the things that docker does with cgroups to get that to work.And so even though it works, that configuration, it's kind of dangerous in the sense that it can lead people to break other stuff without realizing. And so that's like, yes, we should support this, but not this naïve approach that I did.ADRIANA: Yeah.LUIZ: Luckily, this is such a sort of common problem that we'll probably have a better cgroup handling in a future release. And once we got that, then we'll be able to support that feature. But for now, it's on my sad, unmerged PR.ADRIANA: But it's interesting because it was a good learning, right? Because it was like, hey, we got this to work with this special unmerged PR. But then it kind of led to more questions. Right. And I think this is a really great lesson for anyone. Whereas, yeah, this might seem like, oh, this is an easy solution, but what are the repercussions? And that's why pull requests exist...LUIZ: Exactly.ADRIANA: ...so that we can mitigate against weird things happening, because you just simply do not know what the side effects are going to be from, like, oh, I added this little flag. What's the worst that could happen, right? So, anyway, it was a really cool side project. I'll provide a link to the blog post where I detailed our adventures in the show notes. And then the final thing that I wanted to talk about, because when you were On-Call Me Maybe, it was one of the reasons why we brought you on was to talk about how you had played around at one point as part of a hack week to try to add OTel instrumentation to Nomad core. And this was, I guess, over a year ago. So I was wondering if you could talk about what you've learned a year on what the status of that is right now.LUIZ: Yeah, cool. Yeah, that was still pending. I would say it's a very side project of mine, just like an exploration. I think I try at least three times now, try to get some OpenTelemetry to Nomad. And every time I learn something new, which is great and sort of like, builds on top of the previous attempt. So, like, a bit of history is, like my first attempt was sort of a very big view of, like, I want to instrument whole Nomad. I want to be able to create this sort of trace and spans from, like, I submit a job, that job gets scheduled, that gets picked up by a client, and that client starts a test driver, and the test driver calls Docker to start.I wanted the whole flow as trace and spans and all of that. That turned out to be a terrible idea just because I want to say it's not doable, but it takes a lot of code changes to get to that point. So that's the first learning. Don't try to do everything at once. And then my second attempt was more focused around not exactly OpenTelemetry in Nomad per se, but like helping people using OpenTelemetry and running things in Nomad to get information. Now I forgot what the name of that component is, but it's like a way for if you have an application that uses the OpenTelemetry SDK to automatically pick up information from Nomad, like the allocation ID, the job name.So you use the OpenTelemetry SDK. There's like environment variables that are sort of standardized.ADRIANA: Yeah.LUIZ: So provide those things automatically.ADRIANA: Oh, yeah, because I think there's, like a similar thing in Kubernetes where you can automatically grab from your Kubernetes pods.LUIZ: Yeah, there's a whole spec for that. I forgot the name, how it's called, but it's a way to sort of automatically infer information from different sources based on either environment variables or API calls. So I kind of hack around that and sort of works. There's another set branch that I didn't emerge with this work, but the challenge there is sort of like it's kind of hard to tell what information is relevant because you also don't want to shove a bunch of things because it's going to increase your network packet size. It's going to generate a lot of extra information that you may not care about. So I'll have to build a way for you to customize which information you want. So that's where I put a pause on that.And then my last attempt after talking with you, Ted, and some other folks on the OpenTelemetry community, I learned that don't try to boil the ocean, don't try to instrument everything at once. Focus on your core business logic. Start there and then you're going to get a lot of value from that already and then you can start building on top of that. And so my latest approach was like, okay, I cannot create a whole trace. I cannot create that relationship between traces, but can I use metadata to connect them? Probably I should explain this, but one of the challenges with OpenTelemetry in Nomad is that OpenTelemetry, and more specifically the distributed tracing aspect, is sort of focused on microservices and network requests and sort of keeping a track of those network requests.But in Nomad, the complexity is sort of built in into the Nomad binary. So the complexity comes almost like from local function calls rather than network requests. And so if you try to create spans for function calls, you get like tiny traces of a few milliseconds that are not really useful and it just generates a huge overhead. But what helped me there was understanding this notion of like, oh, I don't actually need to connect the traces per se. If they have the same metadata, then sort of like whatever platform you're sending those traces to, like Lightstep, Honeycomb or Zipkin, Jaeger or whatever, then you can start querying traces that have the same metadata so you don't have an explicit connection between your traces.But the metadata becomes a way for you to start to understand what happened. And so that was the last attempt that I did and it was quite successful. It works very nice in terms of trying to understand the inner workers of the especially the Nomad scheduler, because that's sort of like the magical box. You run a job and suddenly you have a bunch of allocations for who knows what reason. And so my goal was trying to understand what happens in there. Because if you look at the source code for Nomad, people that know Go, who like to get an adventure, search for a function called compute group in Nomad's GitHub and try to understand the function.And then come explain to me once you understand, because that's the function that gets a job and generates the allocations. So it looks at the clients, looks at what allocations already exist. And it's sort of like the central point of all Nomad features, more or less. I think people don't realize how many features Nomad have, but things like preemption deployments, disconnected clients, all of this sort of needs to take it into account when you are scheduling things and it all comes into that function.So like, Compute Group is my nemesis, and every time I need to touch that, it's like I need a fresh cup of coffee to go there. But yeah, my goal is like, okay, can I make this function more understandable using telemetry? And it helps in quite a bit in some ways, but there are things that this process is just complex for this. You kind of need to embrace that sometimes.ADRIANA: It's interesting, right? Because you start projects like this, you're like, of course it's going to be easy to instrument.LUIZ: Yes, there's an SDKADRIANA: It's like the k0s on Nomad thing. Of course it's going to be easy. And it's like no.LUIZ: There's tutorials, and there's all these different materials. Just go install this SDK. But no, it's a very different use case, right? Normally you come from this microservice architectures and then you're trying to instrument the communication patterns between them. But I'm trying to do something very specific that I don't think would apply to most people using OpenTelemetry. So, yeah, it's almost like...ADRIANA: And you said a lot of the processes are asynchronous, too, which makes it kind of work with, right?LUIZ: Yeah, so like when you have...the lifecycle of the job, right? Like a new Nomad job run that generates a HTTP request to whatever client or server you're talking to. That request needs to go to the leader, so there's another request going to the leader. But then once it gets to the leader, then there's a bunch of asynchronous stuff happening. So it creates an evaluation. That evaluation gets picked up by what we call it a worker, like a scheduler worker that does all the computing. Once it figure out which allocation needs to get run, then a client picks up. There's no direct network request that covers the whole thing. It's a bunch of put in a queue somewhere, put in a broker somewhere, and then that gets picked up. So that's sort of when you lose your trace a little bit.But the tricky thing about Nomad is the network request is like the easy part. The complexity is like, what happens after you receive that request. So that was the thing that I wanted to instrument, was like, yeah, network requests, yeah, they happen, it's fine. I know who is talking to who. But inside each process, that's sort of like where the challenge lies. Like, okay, how do I get visibility and what's happening right now in there? And that's like, I don't know, it sounds like a fourth pillar, perhaps. We have metrics, logs, and traces. Maybe there's something new that should exist there. But yeah, that's challenged it's like understanding what's happening inside the process.There are a few tools for debugging, like DProf and things like that, but they're very low-level in a sense and you don't always want to run those sort of additional instrumentation in production. That was the challenging part.ADRIANA: Yeah, it I did see something that came out this week where there's, like I want to say there's, like, some sort of go auto instrumentation air quotes, maybe not air quotes with eBPF that can help give some additional insights where that could be a game-changer.LUIZ: Yeah, that would be pretty cool because eBPF, you can sort of hook into anything like any sort of system call or whatever that combs from your program. Yeah, that could be interesting. Specifically like, thinking of the Nomad case when the scheduler is very complex. But there's also a lot of complexity in the client because, oh, I need to run a Docker container. Cool.But it's not just that. Especially in Nomad, we have templates, artifacts, volumes. So you need to mount a volume, download a file, you need to render a template, you need to fetch tokens from console involved. So running a simple container, there's like a whole lot that needs to happen beforehand. And we call those like lifecycle hooks. So you can have things that happen before the task starts, things that happen after the task starts. And a lot of those interact with the operating system. So being able to instrument sort of like what's the Nomad agent trying to do against the OS could be very nice.ADRIANA: Yeah, cool. I think there's definitely more work to be done in that area. But I'm glad that you've continued experimenting. Even if it's not gone, maybe as far as you would like, I think it's still progress, so, you know...LUIZ: Yeah, it's all learning.ADRIANA: ...it's awesome. That's awesome.LUIZ: Like, I think it helps something different, I guess. Something different to learn something different. It's always good to keep up to date what's happening. And a lot of people are starting to adopt OpenTelemetry more. So even if it never comes, that OpenTelemetry is integrated into Nomad core...But I think it's helpful to at least understand because my target audience will maybe use OpenTelemetry on their stuff. And whenever I talk to them, I sort of need to understand what they are doing and how things work. I know if somebody comes and open a niche and say, oh, I'm trying to run the OpenTelemetry Collector in Nomad, I would need to know what they mean. And having this sort of exploration is very helpful.ADRIANA: Absolutely. Cool. Well, we have come up on time. We could keep talking about this forever, honestly, so we'll have to have you back again. Thank you so much, Luiz, for joining today for geeking out with me. Y'all don't forget to subscribe, and be sure to check out the show notes for additional resources and connect with us and our guests on social media.Until next time...LUIZ: Peace out and geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:Renata Rocha has been working with technology since 1997, first as a sysadmin. She then found her passion for Cloud Engineering and never looked back. She has been at Slalom since 2019 and loves everything about the Cloud, Platform Engineering, and the endless possibilities it brings us.Find our guest on:MastodonLinkedInFind us on:All of our social channels are on bento.me/geekingoutAdriana’s X (Twitter)Adriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:International Lefty DayNoYAML.comHashiCorp Configuration Language (HCL)Python (programming language)HashiCorp (Hashi)HashiCorp TerraformHashiCorp NomadSysadmin Appreciation DayMainframeRoyal Bank of Canada (RBC)COBOLJob Control Language (JCL)Platform EngineeringServerlessData centerCyberpunkShopifyBellCentral Office (Telecom)Jacarepaguá (Rio de Janeiro)Rotary dialHow to use a rotary dial phoneDigital transformationADHD hyper-focusPomodoro TechniquePomodoro TimerTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, Reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And today geeking out with me, I have Renata Rocha. Welcome, Renata!RENATA: Hi, nice to meet you. I'm Renata Rocha. I'm a principal of Cloud DevOps security, actually. Platform engineering with Slalom Build. I have been at Slalom since 2019. I love my job, I love what I do, and I love technology.ADRIANA: Awesome. And where are you calling in from today?RENATA: I am based off Toronto, Canada. Like Adriana. Okay. Yeah.ADRIANA: Awesome. Yay fellow Torontonian, fellow Canadian and fellow Brazilian.RENATA: Fellow Canadians and fellow Brazilian as well. Yes.ADRIANA: All the things. All the things. Cool. Okay, so we're going to start up with some warm-up questions. First question, lefty or righty?RENATA: I'm a lefty. I'm a very proud lefty. I do everything with my left hand. My right hand is absolutely useless.ADRIANA: You're the first lefty that we've had, and I'm so happy to have a fellow lefty on the show because I, too, am a proud lefty. And International Lefty Day is on August 13.RENATA: I did not know that. That makes total sense. By the way, awesome.ADRIANA: Super excited for Lefty Day. I always forget about it until after the fact, so I'm hoping I'll observe it this year. So lefties unite.RENATA: Lefties unite.ADRIANA: Okay, next question. iPhone or Android?RENATA: iPhone. It's interesting because I actually would prefer an Android, but it's been a while since they released a very small flagship Android phone. All the flagship Androids are huge, bulky phones and I have very tiny hands. Okay. So I have been an iPhone user because it's the smallest flagship phone I could find in the market. I would go back to an Android any day if they release a smallest Android flagship phone. Okay. Just because of that.ADRIANA: Yeah, fair enough. I actually miss the really small phones. Do you remember those little tiny Nokiavphones that everyone used to have?RENATA: Absolutely. I love those. Oh, yeah. They were so cute, right? Yeah. I'm under five foot tall. Okay. I'm very petite, so my tiny hands would rather have a small phone that fit into just one hand. I don't have, like, two hands to type. Yeah. They don't think about people like me when they design phones.ADRIANA: Right? I do feel like we're outnumbered by taller people here in Canada. Awesome. Next question. Mac, linux or Windows?RENATA: Oh, great question. I love Linux, but I have been a Mac user for a number of years. I think Mac offers a great desktop environment with also a great Unix like system. So I can do everything in terms of programming development on a Mac with a pretty environment in front of me. But for systems, my setup, my servers, they are all Linux, obviously. Windows is a no, no, I don't touch that. Okay.ADRIANA: I'm kind of the same with you there, actually. I cry every time I have to touch a Windows machine. I'm sorry, Windows people, don't mean to offend, but it's just not my thing.RENATA: Oh, my God, I feel no, no, I feel dirty. Okay. Yeah. Awful. Ewwww...ADRIANA: Next question. Favorite programming language?RENATA: Python. Absolutely.ADRIANA: Me too. I love Python.RENATA: Python is beautiful. It totally makes sense. I spent many years thinking, like, oh, my God, I hate programming. I hate programming. And then one day I discovered Python and it was like, okay. I love programming.ADRIANA: Yeah, I agree. I feel it's like a pleasure to program in Python. It always makes me happy whenever I touch Python code, so I totally get you.RENATA: Yeah.ADRIANA: All right, next one. Dev or Ops?RENATA: Oh, Ops. Yeah.ADRIANA: I kind of figured.RENATA: I come from a sysadmin background. Okay. I feel like inside of me there is a sysadmin. I will always be a sysadmin Okay. So Ops any day.ADRIANA: And that will tie right nicely into our topic of discussion in a little bit. Okay, two more questions. I think I know the answer to this one. JSON or YAML?RENATA: YAML Yeah. JSON is weird, and I think you can do pretty much everything with YAML. YAML is just easier, but I don't know, that's tough. Yeah, you kind of have to do it. That's my favorite thing. Can I just say HCL?ADRIANA: Oh, yeah, I'm right there with you, actually, because I always tell people I feel like HCL is the love child of JSON and YAML, because I think it's got the nice organizational stuff of JSON without the clutter, which is what I like about YAML. It's uncluttered. So, yes, I am totally there with you.RENATA: It just works, in my mind. The first time I saw HCL was like, yeah, this is right. This just rings a bell. And yeah, I love it. I think it's beautiful.ADRIANA: Totally, totally agree. And for those who are not familiar with HCL, I think it stands for HashiCorp configuration...RENATA: HashiCorp configuration language. I think that's what it stands for. Yeah.ADRIANA: Yes, that's right. Which is used for all the Hashi products, which is super awesome. All right, final question.RENATA: Sorry...ADRIANA: Oh, no, go ahead, go ahead, finish the train of thought.RENATA: I was just going to say that I'm a HashiCorp fan girl. Okay.ADRIANA: You know what? I have become too, mostly. I know, like, you work with a lot of Terraform. For me, my HashiCorp fangirldom comes from working on Nomad, so I can totally relate.RENATA: Yeah, Nomad, very underrated product.ADRIANA: Very underrated. Totally agree. And maybe we can talk about that a little bit later as well. Final question. Do you prefer to consume content through video or through a blog?RENATA: Yeah, a blog. Absolutely. I like reading stuff. Okay. The video things...sometimes I'm on the bed and it's late at night and I don't want sound, I don't want to see things. And on a blog, I can just read at my own pace. Yeah, I prefer reading stuff.ADRIANA: Yeah, I'm the same way. Of all the people I've asked so far, everyone is like, blog over video, so I've yet to encounter a video person. I know they're out there but yeah, I've yet to encounter one.RENATA: Yet you're making a video.ADRIANA: But this is also going to be out in audio. So I guess then there's the question: audio versus video. I'm more of an audio person. Because then I can...I'm a podcast person, so I like to walk and listen to podcasts.RENATA: Um, I like video. I need the visual cues. I don't...podcasts feel weird for me. I need to see people talking.ADRIANA: Yeah, I can appreciate that. What I do like is video with subtitles.RENATA: Oh, that is perfect. Video with subtitles. Yeah, that is...see, because you can read and then you can also see people's faces. Yeah.ADRIANA: Yeah, I'm totally there with you. Cool. Well, let's move on to our main topic because, as you pointed out earlier, today is International Sysadmin Day. From your background as a sysadmin...RENATA: Today is July 28th. It's the last Friday of July. It has been International sysadmin Appreciation Day since the year 2000. Like I said, I have been a sysadmin in the past. Then I moved to Cloud engineering, platform engineering...how we are doing it today. And I feel like these practices, they wouldn't exist if it wasn't for the sysadmins of the early days. We wouldn't have these things if it wasn't for systems administration. So Happy Sysadmin Day. Although we won't be seeing this video until later, but yeah, Sysadmin Day.ADRIANA: Because you said Ops over Dev...What is it about sysadmin that you love so much that you think people really underappreciate?RENATA: The thing that people don't see is when you are a very good sysadmin and you know what you're doing, your work is supposed to not be seen. Okay. The sysadmin is supposed to be invisible because the system just works. And the sysadmin is someone that is not there because you don't need to see them. You only notice your sysadmin when things crash, when things are wrong, when things are broken. So if you are good at your job, you're not appreciated, you are not seen, you are not present. Okay? So that is the interesting part of being in a sysadmin. But it's very satisfying to look at your uptime and see that your systems are up, that things are working, that things are just fine. Okay. I love seeing all my monitoring statistics and seeing yeah, everything is so finely tuned, everything is working so fine.That ties a little bit into observability as well. Observability. I know you are an Observability person. Okay. So, yeah, like I said, everything I kind of branched off and became like a specialized feature of all the things that we used to do in systems. And today, of course, how computers evolved and how we do things these days. They became their own specific practices and fields in their own but I still love...I have this passion for the way it's a type of nostalgia in a certain way, of how we did things in the past. And this appreciation for people who are still managing systems, especially more old school systems are still in operation.I know there are people who still manage mainframes these days and that's such hard work. Okay. It's amazing. Oh my God.ADRIANA: It's a very specialized skill.RENATA: Yes. And you don't find many of them. Okay. Yeah.ADRIANA: Yeah. As a student...I don't talk about this much because it was very traumatic...but I had a summer job working in the mainframe department of RBC.RENATA: My God.ADRIANA: It was something...it was something...I will say in air quotes, a COBOL code change that it was like, I don't know, I think it was like commenting something out or whatever. And it took forever to push that through because you had the Job Control Language at the time. So the language to compile the language. And I was like, oh man. No, not for me.RENATA: As someone who has also touched some COBOL calls back in the day...Yeah, I understand what you're saying. Yeah, I have some feelings. Like I said, nostalgia. Not necessarily want to go back there, but I'm going to say that I have played with some virtual instances running systems just for fun. Okay, yeah,ADRIANA: Very cool. Very cool.RENATA: Very cool. Yeah.ADRIANA: So, as sysadmin, what do you think has been like the biggest change that you've seen over the years as technology has evolved? As your career has evolved?RENATA: Sometimes I talk to people that I interview that are still working in systems and they are moving towards platform engineering. And what I think of what I used to do, and the biggest paradigm shift is that you used to log in directly into the machine and debug whatever was going wrong and fix it. So you had this control over each machine that you were running. You were very close to the system. To be quite honest with you. That feels very good. It gives you a sense of control.And with Cloud and platform engineering, you don't do that anymore. You simply don't log into the machine anymore. And you have so many layers of abstraction, especially if you're running something like serverless code, something is broken, something is wrong. You just fix that on your Terraform code, for instance, and then you redeploy it. So you don't even go to the machine anymore. You just fix your code and you redeploy it and then it's back up online. So it is a different way of thinking on how you were going to troubleshoot the problems and how you were going to deal with whatever is running your code. So it works differently.It's a lot of abstraction on top of it and you are not so close to the server as you used to be and you are probably going to miss it when you start working with so many abstraction in front of it. Okay, I understand that, and like I said, I miss it. It was fun. But at the same time, it gives you a lot of power because you had to go machine by machine to fix the problems and now you can fix the problem in hundreds of machines at the same time. And that is so amazing.ADRIANA: It's so true. Yeah. Do you think that having been an old school sysadmin where you were physically touching your machines...do you think that gives you a better appreciation then for the types of things that are happening in the Cloud versus the newer folks coming into it who have not known sysadmin in that same way, where you're, like, sometimes going into a data center and physically touching a machine, do you think that gives you an advantage as a result?RENATA: I hate saying better or advantage, but it gives me a different perspective. Okay. I work with a lot of people who have never been inside a data center in their lives. And when I tell them that, yeah, I used to work inside a data center and it's freezing cold, it's weird, there are like these noises and it's kind of soothing, the noise of a data center. Okay. You just have to wear a jacket because it's so cold and they're like, "Oh my God, did you work at a data center?" Like it was the craziest thing they've ever heard and yeah, it was a great job, by the way. I liked it very much.Yeah, I would ride my bike to the office, if the data center was actually downtown. I would ride my bike there and then work there all day. We didn't have Windows because it was a data center. Felt like this box, like, surrounded by computers for hours. And then I would ride my bike back home. Yeah, great job. Loved it all. Lots of cables. It felt so Sci-Fi, so cyberpunk.ADRIANA: If anyone ever has had an opportunity to work at a data center or tour a data center, it is surreal.RENATA: It is surreal. Okay. It feels like living in the future. Okay. So disconnected from life. Like all the people walking around living their lives, and then suddenly you are in that box of computers. Right? Yeah. It's amazing.ADRIANA: And you're holding the keys to the kingdom...to the computer kingdom, because you've got the admin passwords, you could do some serious damageRENATA: Yeah.ADRIANA: As an Admin, even as a Cloud engineer. Same sort of thing. But I think there's like a different...it's a different feeling.RENATA: It is a different feeling. You have this power that like yeah, if you do something wrong, a lot of things go wrong, but at the same time, you feel so isolated from everything else that is around you because it's this black box, so completely separated from the life outside.Like, I have the story that I was at the Shopify data center and there was a city-wide blackout that lasted maybe a couple of hours. And I didn't know it was happening because the data center has backup generators directly connected to whatever. And until someone told me, "Did you know there's a blackout happening outside city-wide?"I was like, "No, I didn't realize that." Yeah. The entire city has lost power. Okay. Amazing. That's why I like working there.ADRIANA: It's so true. I have a similar story because when Toronto had that big blackout in 2003, I think it might have been before you moved here, I believe, right?RENATA: Yeah. It was before I moved here. Yeah.ADRIANA: So in 2003, we had that massive blackout that lasted like a day and a bit. And I was working at a client site. The client at the time was Bell, and we were in the building next door to their Central Office. So the central office where they keep all their network equipment and stuff, like phone stuff, right? And so because we worked next door to the Central Office, we had the backup power as well, because the buildings were attached. So when there was the blackout, the lights blipped very quickly, and then it was all good. And so we're, like, keep on working. And then at the end of the work day, we start getting messages, like, there's no power in Toronto. We're like, "What?" Meanwhile, there's, like, power in the building. The elevators were still working. Everything was fine. And then you go outside, it's like complete chaos. Everything's out. Traffic lights, subways. Yeah, so yeah, the nice little bubble.RENATA: The nice little bubble. Okay. And that feels very comforting. And like you said, it feels like a lot of power that you are in this place that has everything. It has like the fastest Internet access you can imagine. It has power even when the entire city is in a blackout. Yeah. It is fascinating. It is such a great experience. And although I don't feel I have an advantage or I feel better than the people who didn't have this experience, it gave me a different perspective. I feel like I have learned what I do today at a slower pace. And I saw the Internet growing since the very early days of the Internet, and I was inside. What was the Internet back in the day?ADRIANA: Yeah. So true. Such a unique advantage that's something like, people from our generation, I feel like it's something that folks like, my daughter was born in 2008. She's grown up in a world with computers. It's ubiquitous. Smartphones. This is stuff where we're growing up along with it. It's like, oh, internet. Oh, cell phones. Oh, smartphones. We're all connected. Whoa, this is weird. Like, social media was not a thing when we were growing up. I mean, good luck trying to reach somebody. They're not home, you can't reach them. And you had to know people's phone numbers.RENATA: Phone numbers? Yes. When I was a kid, we didn't even have phones. Landlines so easily this way. And a lot of people didn't have landlines. My husband only had a landline when he was like ten years old. People had to write letters to each other. Imagine that.ADRIANA: I had a pen pal when I was growing up, and I totally remember my mom used to tell me stories of when we were living in Rio....I was too young to remember...but she was saying there was for a time we didn't have a phone yet, so she had to use the neighbor's phone to make phone calls. It was not the easiest thing to procure. It was, like, kind of a process to procure a phone line. And these are things we take for granted now.RENATA: Yeah. Where my husband used to live, which was a neighborhood in Rio, Jacarepaguá. They had this phone in the center of the neighborhood where people used to go and make a call because people didn't have landlines where they lived. Yeah. Crazy. And you think of it like, oh, must have been like a very rural area. No, it was just like a normal neighborhood, actually. Kind of an upscale neighborhood. It was just because landlines were not everywhere back then. Okay. Not a lot of people had that.ADRIANA: And I remember rotary dial phones when I was growing up in Rio, too, which, when I came here, i's like, "What? It's not rotary dial phones?" I still have fond memories of those.RENATA: Can you hear the noise? (of rotary dial phones) I saw something fascinating recently. I went to a doctor's office here in Toronto, okay? And there was, like, this emergency phone, in the doctor's office, and it was a rotary phone, and there was a sign next to it teaching people how to use a rotary phone.ADRIANA: So cute. The things we take for granted. We're like, "Of course! That's how you use a rotary phone. And people are like, "Uhhuuhh..."RENATA: Yeah. Wait for the dialtone. After the dialtone, you put a finger on the number and to wait for it to come back. Good times.ADRIANA: Good times. I kind of miss rotary dial problems. Yeah. Fun memories. Fun memories.RENATA: They were cute. They were cute. Yeah. Yeah. We had a red one, like classic red rotary phone one.ADRIANA: Oh, that's adorable. It's better than my beige one.RENATA: Amazing. It was adorable. There was also the beige ones. They were also very classic. Yeah.ADRIANA: Yes.RENATA: But yeah, it's funny to see how things evolved, and they evolved very fast. And I think the beautiful thing about working in technology is how you see things going from, like, the old rotary phones and then the dial tone phones and then the Internet, which leads us to the different fields, the specializations we have on platform engineering, SRE, etc. And now the new field of AI that people are like, oh, my God, AI, etc.It's just a new field, okay. And we need to learn how to deal with this new tool, this new technology, and how it ties together into the other things that we do. And that's the beauty of it. Right? This is why I do this. This is why I'm so fascinated by technology. I love learning things. I love being immersed in it, and I'm absolutely fascinated about learning whatever new thing, whatever new tool that people bring us today, tomorrow, next year.ADRIANA: Yeah, I totally agree. It's interesting because I'm the same way. I'm like, "New technology. Giddy up. Give me more!" I want to learn more about it. And yet some people are so terrified of technology, and we see this across our industry. We see it especially, like, in large corporations, whenever there's big transformations, whether it's a digital transformation or DevOps, Agile, whatever, even bringing in Observability...there is such resistance to change that can get...it legitimately freaks people out, right? Because to a certain extent, I think we're creatures of comfort, and we like what we know and understand, and having to deal with something that's a brand new learning curve can be terrifying.But what kind of advice do you have for people who find themselves in this scary place where they're resisting change.RENATA: Try to find yourself in a place which is 50% comfort and 50% challenge. Okay? That's a very good measure of what will bring yourself still that cozy space where you feel, yeah, this is good. This gives me this warm heart, this warm embrace of things that I'm comfortable with, but also doesn't make you bored, because once you get bored, you don't have anything to look forward to. And this is what makes us human, okay? If we didn't have any challenges, if we didn't have anything to look forward, any goals in life, we would still be living in the Middle Ages. We would still be cooking...eating raw food. We wouldn't be working with computers today. And that's what makes us different from other animals, is what makes us special. Okay? Yeah. This is pretty much what I try to tell people that find that sweet balance of comfort and challenge, because also, if you do challenges all the time, you're going to burn out. You're going to exhaust yourself. It's unsustainable. So find exactly how much comfort you can at the same time while finding that amount of challenge that makes you want to do something new.ADRIANA: Yeah, I totally agree. That's such a great approach because that way, as you said, you're not, like, going all in on it, and then that can be super overwhelming. But getting to a point where it's just like, oh, okay, this is interesting enough. I'm going to poke into it and kind of build on it. Iteratively, I think, is a really great way to approach it.And I think in technology, I feel like because, the nature of what we do, it evolves so rapidly, it can be very overwhelming. So I think people get freaked out. Oh, my God, there's another thing that's changing, another thing I have to learn. So I think to a certain extent, you kind of have to just pick a thing that's interesting, that like, okay, I'll dig into this a little bit more, and then dig into that and get into this habit of always learning a bit. And I think what people freak out with is they think they have to learn all the things.I resigned to myself, to the fact that I don't know everything. I'll never know everything. But I surround myself with people who know things, and that's okay.RENATA: You don't have to know everything. It's impossible to know everything. And you will never be an expert on anything because things evolve very fast. And like I said, before, technology is very dynamic. What you know today is never going to be what is the new technology, new big new thing of tomorrow. So what you have to do is to pick something that you want. Okay, go choose. I want to learn Terraform. Something that I don't know.Okay, I know Terraform, but just an example. I want to learn TerraForm. It is something that I'm interested. Just pick and choose and learn a little bit. Don't throw yourself in like, "I'm going to learn everything about Terraform today." I tend to do that.So that's some advice for myself. Yeah, I'm totally like that.So learn a little bit of it today and then give myself some time. Tomorrow I'm going to do something that I'm comfortable with and then learn a little bit extra. Just iterative process of learning so I don't exhaust myself. And also it gives your brain some time to absorb the knowledge. You don't feel exhausted, you don't feel burned out. The burnout is real. I always tell especially young people because they feel they are indestructible and they can learn everything everywhere, all the time.And then sometimes the burnout kicks in and they feel, "Oh my God, I cannot learn anything anymore and I hate this. I will move to a cabin in the woods and never see people ever again."This happened to me and that's why I keep telling people that it will happen if you don't take care of yourself. Please take care of yourself. It will happen. So give yourself some time. Go slowly, slow, easy, and yeah, just pick a subject here and there. You don't have to learn everything.And also no one expects you to know everything because we know it's impossible to know everything.ADRIANA: Yeah, I totally agree. And I want to go back to your comment earlier about burnout, because I agree with you. When you're young...first of all, when you're young, you feel like you're indestructible, invincible. But also, employers take advantage of that, too. They're like, "Well, you don't have a family. You don't have a social life. You just finished university. Of course you can go and put in the long hours."And I think it's so important for younger folks to give themselves permission to not do all the things, to take it easy, to not have to hustle so hard that they burn out. Because the same thing happened to me when I finished school.My first role out of school was so intense that I burnt out really fast and I hated my life. But the only good thing that came out of it was I learned to say no, and I learned to defend myself and stick up for myself in terms of, like, I've had it. Burnout. My brain doesn't work. Need to take a step back. And I think more young people need to do that. Just because you are young, don't have a family, whatever, don't have a partner, does not mean that you don't deserve to have a life.RENATA: Yeah. My suggestion is always to find an offline hobby.ADRIANA: Yes.RENATA: Something that is completely unrelated to your work. Learn to run, to climb, to do Yoga, to paint. Go learn to play the drums and musical instruments. Something that is completely unrelated to your work. You don't need to do some unpaid work. That's what I'm trying to say.ADRIANA: I totally agree. What's your go to activity?RENATA: I like to run, I like to cycle. I do Yoga, I have a garden in the summer. I have a ton of activities that I do that are completely unrelated to work. And they help me not think about work. They make me healthy, they make me happy. They make me a person that is not only the Renata who is at work, the Renata platform engineer. I am a more complete person and I'm also happier. I have a life that stops at 5:00 pm.ADRIANA: That's such great advice. I totally agree. Because of my ADHD, I get the hyper focus, and so I have a really hard time peeling myself away from things, and I tend to obsess over unsolved problems and can't shut it off. And sometimes I have to take myself aside and say, "I give you permission to not think about work, to disconnect." And it's okay to give yourself permission to shut off and pursue your things. Like, life is not all about work. It's going to be there tomorrow and the day after that and the day after that. I think employers are just looking to make sure that you complete your deliverables, that you're reliable. And I feel like if you do those things, then you're in a good position.RENATA: It...I also find that once I started taking care of my mental and physical health after work and not dedicating myself to working 15 hours a day, which I was doing, at some point I became much more productive at work.ADRIANA: Yes.RENATA: Which kind of makes sense if you think about it. But when you're younger, you don't think about it because I'm healthier. My mind is healthier. My body is healthier. So when I start my day in the morning, I can kick right in, start working, do a lot of things. I can concentrate much better. I haven't been sick in years, and I used to get sick a lot more. I just work better if I do this. My hours in a day are in a better person that can do just work for those hours a day. And I was so drained out by the burnout that I was not giving myself fully to work during those during those 15 hours that I was working before. So, there's that.ADRIANA: Yeah. I find also, like, the act of time boxing your day. You're like, holy shit, I got to get all this stuff done by 05:00. So you're like, okay, I'm going to be super efficient, right? Because you want to get as much done as possible, and then it's like and plus you have something to look forward to. At the end of the day, it's like, hey, there's like, relaxation on the horizon. It's awesome.RENATA: Yes, absolutely. When I have to deliver something and I have a deadline, I like to use a technique called Pomodoro. Some people are very familiar with it. You do 25 minutes of work, and then you stop for five, and then you do another 25. And after some four of 25, you give yourself a longer break that allows your brain to process the work that you did some rest. And it makes me feel way more productive because I just focus and just works very well for some type of people. It works for me. And if you're listening, and you are not familiar with the Pomodoro technique, there are some timers you can use on your browser or on your code editor. Give it a try. It might work for you as well.ADRIANA: That is very cool, I hadn't heard of that. But I definitely...like, taking the breaks is so important. And again, ADHD brain is like, you will not get up from this until you solve this problem. But whenever I do force myself to walk away and take a break and I come back, I'm like, oh, shit, I should have done this before.RENATA: Try the Pomodoro. Okay. Because they force you to take a break every 25 minutes, and it's a 5 minutes break. And then when you come back, you're like, "Oh, okay, I can pick this up again." Okay. It's not a long break. It's 5 minutes. Okay? Usually I just do maybe like, some Yoga poses for five minutes, and then I go back. I love Yoga. It's a great thing for my mind. It makes me relax. And then I come back, and then I do it again, and then I keep doing that, and I don't know, I just write code beautifully when I do this.ADRIANA: Yeah, I'm going to check that out because that sounds like something I could use. And also, I'm a huge fan of Yoga. I don't think I'm nearly as advanced in Yoga as you are. My flexibility is crap, but I do enjoy it. It's nice to it challenges your brain because you're so busy trying to hold the poses, you can't think about anything else. So I think it's a lovely way to just unwind.RENATA: Yoga, as with anything...technology...is not about the flexibility. It's about the inner journey, okay? It's about learning, understanding your body and where you are today. It doesn't matter where you were or just where you were going. Think about yourself, your inner body, where your balance is. So don't think about anything else. It's not about flexibility. It's just about the journey. So if you enjoy it, that's what matters.ADRIANA: Yeah, that's actually a really good point, because when I first started Yoga, I see it really mad that I was like, I look like shit doing these poses. Right? And then I'm like it's okay.RENATA: Yeah, exactly. No one cares.ADRIANA: As soon as I got over that...nobody cares, especially when I do it at home, nobody's watching. So it's great. And you start to see some progress. I mean, you're competing against yourself, which I think is probably the most important thing. Are you improving? Are you getting something out of it? Are you enjoying it?RENATA: That's why good Yoga studios won't have a mirror, because you're not supposed to look at yourself. It's just supposed to feel yourself.ADRIANA: Oh, that's cool. Good to know.RENATA: I like it very much.ADRIANA: Yeah, I'm a huge fan. I try to do Yoga, like, once a week, and I feel it too. If I don't do it, the joints feel a little stiffer. I'm like, oh, I think I needed this.RENATA: Focus on the breathing techniques. They really help me when I'm feeling stressed out. When I am obsessing over a problem, I just try to focus, recenter, breathe. And that sometimes helps me solve some piece of code that I cannot point or like some architecture that I'm struggling to design. I have a huge problem using diagramming tools. Sometimes I have the idea on my mind, but I don't know where to position things correctly. And then I stop. Take a deep breath. Okay. Do some breathing exercise. And when I look at the diagram again, I'm like, oh, yeah, here, write this.ADRIANA: Yeah. The power of stepping away, taking a mental break from your work, cannot be underscored. It cannot be underestimated. Awesome. Switching gears a little bit. Well, I guess going back to something that you talked about earlier, the idea of embracing the new technologies that come our way and of course, the new and cool technology that is taking the world by storm now that everyone's talking about and either excited about or feeling threatened by is AI. So what is your take on AI?RENATA: I have a hot take about AI, which is it's not going to take away any jobs. It's going to be exactly like Cloud was a few years ago. It's going to create a lot of new jobs. AI doesn't create itself. It's not actually artificial intelligence. That's just like a cool name for it. There are lots of people working to generate those libraries. You've deployed the code, so it actually requires a lot of people, qualified people, engineers develop that. So it is a whole new field that's open for you. It's fascinating. It's very early days, so yeah, it's going to create a lot of new jobs. So if we embrace it with open arms and open mind, it's exactly like when Cloud was born that people were scared, oh my God, it's going to take away jobs and look at where we are today. So embrace it, learn it. It's great. It's going to be good for us. Just don't be afraid because it's a new technology. It's just a tool. It's not something bad or crazy. Yeah, that's my hot take about AI. It's just a tool.ADRIANA: Yeah, I totally agree with you. I totally agree with you. And I think, like, with any new tools, it can be abused or it can be used to really enhance your job. And I think folks who end up using AI tocheat at their jobs or cheat at work, right? Like using AI to write an essay, you're doing yourself a disservice because then you're not learning. I mean, you lose out in the end. Fine, you get the marks, but you still can't write. Versus using AI as an aid.Like, the example that I like to give is, like, you're writing something, you've written something out, but there's like a character limit. Feed your text into AI to like, hey, can you rewrite this so that it fits within the character limit? I feel like that's a perfectly valid use case for AI, because you wrote it, the concepts are there. It came from your brain. But AI has just taken that little extra burden off of you so that you can complete that last step where you can use AI as, like, inspiration, as a starting point for code. If you don't know a particular language, but you know how to code, but you don't know the nuances of that language. So AI can give you that starting point, but you still have to complete it.RENATA: You can use it as, like, a skeleton generator. Okay. But you still have to refine the results, and you have to analyze it to make sure that the generated content makes sense. So if you don't know if it makes sense, it could have generated something that is useless, something that's bad, something that won't work. So a great idea, like something that I suggest to people is ask AI to generate a recipe for bread. Something simple as a recipe for bread. Try to make that bread. Sometimes it won't work. Okay. Because anything that's bread related, it's kind of tricky.ADRIANA: It's voodoo. Yeah. Bread can be tricky, for sure.RENATA: Yeah, sometimes it won't work. Maybe it will work, but you can't be sure unless you have made bread before.ADRIANA: Yeah, that's so true. And it's interesting, too, because even if you've not made bread before, right? And then you take the AI recipe, try to make bread, it fails.Then you can use that as a springboard to like...but why did it fail? Then you can do some additional research, right? So still need to use your brain there, which I think that's at the end of the day, the important thing, right?Even the AI prompt engineer, when I first heard this idea of a prompt engineer, honestly, I thought it was funny. But it's in the same way that we, as software engineers or ops folks, whatever, SREs when we're trying to solve a technical problem, we're going on Stack Overflow, we're trying to, you know ask Google,like, figure out how to phrase the question correctly in Google, making sure that you're even asking the correct question. And I feel like when working with AI, it's a similar sort of concept.RENATA: If you think about the prompt engineer, which is someone adjusting the prompts they feed to AI to get the correct results, isn't that very similar to platform engineering, adjusting Terraform code to generate the correct results on the Cloud compared to what we used to do as a system in that we wouldgo directly into the machine. That is a hot take.ADRIANA: That is a hot take. I like it. I like it. That is a very cool way of looking at it.RENATA: Yeah. So maybe that is the new job, the prompt engineering, that's a new career path that someone will follow, probably in data engineering, and I'm excited to see what comes of it. I am very open to new tech and seeing what the world brings us.ADRIANA: Yeah, absolutely. I'm right there with you. I think prompt engineering can be very fun. Yeah. And don't be afraid of AI. I think there's some cool things that can come of it. It can really help with our jobs, and it'll be exciting to see where it takes us. I was talking to someone yesterday about AI, and I'm like, oh, could we ever find ourselves ina position where we end up with Skynet? And you always think about these things, but...RENATA: I don't think so. I love terminator.ADRIANA: I love terminator too. I always think of Terminator whenever this AI stuff's coming about.RENATA: It is human enhancer.ADRIANA: I'm like, "All hail our Evil Robot Overlords. Here we go." But I think there are some exciting times. There's some cool stuff to come out of it. AI is a human enhancer.RENATA: I like that. Yeah, it's a good approach. I don't think we are quite ready for Skynet. It's going to take, I don't know, maybe 1000 years for us to reach Skynet level of things. Yeah.ADRIANA: Yeah. Hopefully we won't get to Skynet levels. Fingers crossed. Well, we are just coming up on time, but thank you, Renata, so much for geeking out with me today. Do you have any parting words of wisdom to share with folks out there?RENATA: Yeah, well, just don't be afraid. Embrace new tech. As I usually say, stronger people build a stronger world. And peace out and geek out.ADRIANA: Thank you so much. And y'all, don't forget to subscribe. Be sure to check out the show notes for additional resources and to connect with us and with our guests on social media. Thank you so much for joining us today.RENATA: Thank you for inviting me today.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:Jennifer Moore (she/her) is a staff software engineer at InfluxData with extensive experience in software development, devops, and testing.Find our guest on:X (Twitter)MastodonLinkedInFind us on:All of our social channels are on bento.me/geekingoutAdriana’s X (Twitter)Adriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:TI BASICQuality Assurance (QA)GW BASICArgoFirestoreDatabase schemaOpenTelemetryBull Queue NodeJSOTel CollectorDatadog agentLarge Language Model (LLM)Artificial Intelligence (AI)N+1 querySkyNetSelf-Driving PlanesSelf-Driving CarsGovernment legislation around AIFediverseLetterbook SocialMastodonRubyC-sharpTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, Reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada.And geeking out with me today is Jennifer Moore. Welcome, Jennifer.JENNIFER: Yeah, hi, thank you for having me.ADRIANA: Yeah, super excited to have you on. So where are you calling from today? I am in the Dallas, Texas area. Oh, awesome. I think you're our first person from the South.JENNIFER: Yeah, there are programmers outside of San Francisco.ADRIANA: Nice. So to get warmed up, I want to get started with some lightning round questions that I like to ask all of my guests. So it's about six questions. It'll be fast and painless.Okay, first question, are you a lefty or a righty?JENNIFER: I am right-handed.ADRIANA: All right. iPhone or Android?JENNIFER: Android.ADRIANA: Mac, Linux, or Windows?JENNIFER: I guess, Windows? All of the above?ADRIANA: All right, I think you're our first Windows person. Cool. Favorite programming language.JENNIFER: C#.ADRIANA: Cool. Dev or Ops.JENNIFER: Was that Dev or Ops? Is DevOps the right answer? Because that's what I'm going to choose.ADRIANA: Yeah, no wrong answers. So DevOps works. Yeah. When I asked this one to Hazel Weekly and she said, "Yes," so that counts too. Love it. Final question is, do you prefer to consume content through video or text?JENNIFER: Oh, text usually.ADRIANA: All right, cool. Well, that was it. Short and painless. All right, so let's get into the meaty bits. And I always like to ask my guests how they got started in tech. So what was your foray into tech?JENNIFER: So tech is always what I liked as a child. I liked computers and I took some programming classes at summer camp school thing. That was a weird dynamic, but I did that a little bit and then my high school offered some programming classes. I took those and then I went to school to university for software engineering. And after I dropped out of university, I got a job in QA, moved very promptly into QA automation,ADRIANA: Oh, that's awesome. So out of curiosity, what was your first programming language?JENNIFER: My very first programming language...I don't know...If we're going back as far as I can, then like, TI Basic.ADRIANA: Oh, nice.JENNIFER: Yeah, I think the first one I spent any real time with was C.ADRIANA: Yeah, my OG language was GW Basic, but I don't recall doing anything super damaging with it. So my real real one was QBasic. Good times. Good times. Cool. Okay, so more on the QA thing. What brought you on the path of QA initially in your career?JENNIFER: Honestly, it was the job that I could get. It was 2009, so not like the best time to be job-hunting. And I had been looking for things for a while and I got an offer for QA role, so I took it.ADRIANA: Nice. I got my professional start in QA as well. It was not the QA automation stuff. It was the clickety click, fill out Excel sheets, and sit and wait for the developers to fix bugs. So, yeah, I can definitely relate. And then after a while, I kind of begged my manager, "Please let me write code!"So was QA automation kind of the natural thing for you? Because obviously, you're software-minded, so that felt like is that what led you to it from there? From like, QA?JENNIFER: Yeah. I have some experience with software. I had done some part-time programming and obviously I was going to school for software engineering and my hiring manager at the time had a manual QA process, whatever, but was looking to set up an automation, I guess function for it. And so I think she saw my showing up as an opportunity and went from there.ADRIANA: Oh, that's awesome. Yeah. I feel like QA automation is so developer-minded because we're so tired of I think developers are lazy and I see that as a perfectly awesome thing, right? We do not want to repeat things over and over and over again if there's a way that we can shortcut it through code. So I think it's so awesome that you took advantage of that opportunity and put the developer laziness to good use, right? Very cool. Now what kind of work are you doing now?JENNIFER: So now I work at Influx Data. I'm a staff software engineer on what we call the deployments team, which is a little bit of an unusual charter but basically DevOps, like platform engineering work, and helping...The thing that we most definitely are responsible for is like our CI/CD pipelines, maintaining the good health of Argo and the automation that generates our Kubernetes manifests that we're going to deploy, and things like that.As well as a lot of things around development tooling and some infrastructure work and a lot of whatever else comes up.ADRIANA: So that sounds like a pretty good breadth of responsibilities. Because usually you see like, it feels like the description of your work is like a combination of what you would see in a DevOps team these days, plus an SRE team, plus a platform team all rolled into one.JENNIFER: Yeah, and so we do have an SRE team and they take more of the infrastructure than we do, but we work very closely together because we're using a lot of the same tools and using those tools on a lot of the same things. And so that's a blurry line.ADRIANA: Yeah. Cool. And so as part of your job, do you get to be on-call?JENNIFER: I haven't been much in this role. I expect that I will eventually have some on-call responsibilities but the team is kind of in a rebuilding phase and so there's just been an understanding that we don't have enough people to support an on-call rotation and so we'll get to things like during our working hours.ADRIANA: Got you. Now, how about in previous roles? Have you had a chance? Have you been on-call in previous roles? And what was that like?JENNIFER: Yeah. So in my previous role...this was at a company called Screencastify. They make a screen recording product and I was leading the DevOps team there and I spent a lot of time on-call there. I think in particular, we were building kind of a v2 of the platform, which included migrating data from Firestore into a grown-up database with schemas, and that was going about like these data migrations do. And then we had some staffing disruptions where several people who were very senior and critical of that project resigned somewhat in protest of some management behavior and then the whole thing kind of collapsed and I was on-call to support that.ADRIANA: Oh, yikes.JENNIFER: That was a rough month.ADRIANA: Oh my goodness.JENNIFER: But it was just for a month. Um, I understand that the team is still dealing with, you know, like, the after-effects of that, but I'm not a part of it.ADRIANA: Oh my goodness. That's got to be so super stressful in that situation. How do you deal with because it takes its toll on your mental health eventually, if not right away, given not just being on-call, but stresses of changes in your team. So how did you cope during that time?JENNIFER: It I feel like I handled it pretty well. Like I was kind of, I don't know, like I had a sort of active, ongoing incident that I had to continuously respond to for a long time there. And so I kind of just had to do that. And I think having being able to do things that I could do in a self-directed way and things that were obviously important and necessary and I could just do them without having to go through a planning process that you would do for future work was actually kind of helpful for me in that regard. I could just put out fires and I didn't need to worry about the politics that had led up to that situation.ADRIANA: Right, so you're kind of shielded or at least you're able to work kind of in a little bubble to shield yourself from some of the crap so that you could focus on the task at hand.JENNIFER: Yeah. And there was stuff that I had been wanting to do and now there was this emergency and I took advantage of it and I put in a lot more tracing and monitoring and made some application changes to make the whole thing more observable in general. And that was nice. Getting to just do Observability and reliability work as its own dedicated priority was a really nice side effect of that otherwise unpleasant situation.ADRIANA: Oh, that's so cool. Yeah, it's very interesting because I think a lot of times organizations, when they're embroiled in the royal dumpster fire of production shit storms, it's like such a reactive mode. But to be able to...I think it's so cool to be able to take advantage of a shitty situation and basically say, "No, I gotta do this so that we can improve the overall reliability of the system." Is really cool because I think many organizations would sort of almost not be in support of that, in spite of the fact that that's probably exactly what they would need to do.JENNIFER: It's like unpleasant in the moment, but it is very powerful to be able to say that if you want the system to work at all, then we have to make it work reliably, because right now it just doesn't and it isn't. And that's the problem.ADRIANA: And so how did folks like management and whatnot react once did they...Did they start to see the benefits, I hope, of all the wonderful things that you did around improving reliability and Observability?JENNIFER: Um, I think so. So, like, my sort of direct management chain had been pretty on board with the notion of improving our Observability and making reliability a priority. And so I didn't have to fight very hard with my manager or my director. But then once you left the engineering organization, that was when that sort of broke down. And the rest of the organization was very...or, you know, the rest of executive team was very focused on features and deadlines and just delivering things that they could sell to customers. And they didn't view a reliable product as being on that list for some reason, which is always an extremely weird view to take to me, because if the product doesn't work, then even if you can sell it to someone, they're not going to keep paying for it. And so why? I don't know.ADRIANA: Yeah, I totally agree with you. It's interesting, though, that you I mean, it's kind of shitty that you ended up being a situation where afterwards management didn't really see the value in what you were doing. But from a hindsight perspective, it's interesting to see at the same organization with leadership changes, what happens when you have leadership that's fully supporting this idea of, "Hey, let's make sure our systems are reliable."So supporting an Observability and reliability culture versus an organization...within the same organization, it's change in leadership saying, "No, that's not our priority." It's an interesting experiment, and I'm sure aside from the obvious things that we would think, like, yeah, that's obviously not a great idea, but getting to experience firsthand what that was like, I'm sure must have been a very interesting and unique vantage point.JENNIFER: I think it was definitely interesting. I very definitely learned a lot from it. I'm still kind of, I don't know, like synthesizing what that is so it would be hard to teach those lessons to someone else but I certainly came away with a lot of experience.ADRIANA: Silver lining, then.JENNIFER: Yeah.ADRIANA: So out of curiosity, when you're talking about bringing Observability into the picture, what did you do in terms of Observability?JENNIFER: Yeah, so like in that specific case, there were a couple of things. There was like the system had a main sort of application web server that would handle the bulk of talking to clients and ingesting video and things like that, and then a task system that would do some processing of video and analysis and things like that. And I could not see at all what was happening in that task system and I really needed to to understand what was going on with the whole system. And so I basically just stopped doing other things for a few days and wrote up a OTel instrumentation for the library that powered it, which is Bull Queue. So now there is one of those and I wrote it and I did so in anger.ADRIANA: Angry coding. Awesome.JENNIFER: Yeah. And so like that like that actually that was very helpful, probably in ways that my management did not appreciate because it illuminated a lot of areas where the problems were not occurring and the problem was basically database-related.The question was like what is causing all the stress on the database? And the other thing I did was split up all of the database access into multiple different accounts so that I could actually tell the difference between whether traffic was coming from the web service or the task workers or some migration jobs or whatever else was happening. And between those two things I was able to develop a basic understanding of what the problem was.ADRIANA: And then what did you use for visualizing your observability data? Did you guys use something that was like a SaaS product or just something that was hosted internally?JENNIFER: Yeah, they had been using Datadog and so kept using it. That was a decision that was made before I joined the company and so that just was the one that we stuck with.ADRIANA: Fair enough, but it did the job well enough, I guess, with the data that you were receiving.JENNIFER: Yeah, I mean, we could definitely answer the questions that we were trying to answer once we started sending them the data that they would need to do that with.ADRIANA: That's awesome. Do you know if anyone is still taking advantage of the Observability setup that you put in place?JENNIFER: I understand that they are when I left, one of the things that I had been wanting to do or was actually starting to do was move to the OTel special Collector rather than Datadog's proprietary one so that we could experiment with different back ends and things. And it seems like that work continued because last I heard they had moved to Honeycomb, went off to Datadog.ADRIANA: Interesting, that is very cool that the stuff you put in place continued, I'm sure that you feel really great about that, to have a little legacy.JENNIFER: Yeah. I wish it was a happier legacy, but I am glad that it's helping the people who are still there's.ADRIANA: Well, switching gears a bit, I know that when we were chatting earlier, you were talking about you had some thoughts around how engineers learn. So I was wondering if you could share a little bit more on that.JENNIFER: Yeah, sure. So this is this kind of all like this thinking that I've been doing about it kind of comes out of a lot of the sort of public conversation that's happening around like AI and LLMs and such. They're used as developer tools. And I think one of the areas that doesn't get talked about enough in this regard is that the only things that those kinds of AI development tools are really good at doing are the same tasks that humans need to do in order to learn about the complex systems and programming and the systems that they work on. And so it's a little bit of technology eating its own seed corn here when we push these tools. Because it might be convenient for senior people who already have all of those knowledge and skills, but the next generation of engineers who we should be looking out for are just losing all of these opportunities to do really good basic learning work to computers that can't even really learn from it.ADRIANA: Interesting. So you're saying that these AI tools are almost like hindering how we learn as a result?JENNIFER: Um, yeah, kind of. I mean, I think that there like there is a lot of danger that those AI tools can take all of the good, you know, like all of the good learning tasks and, you know, I guess like jobs and roles that people would learn from.ADRIANA: Right. Yeah, I think even to a certain extent, using I feel like there's a fine line, right, like, of like leaning too hard into AI and then but using it as an aid, right? For example, I've had in a couple of cases where I've written something down, I need to summarize I summarize something, but it needed to be like 300 words and I'm like, at 350, I'm like, Shit. Sometimes it can be really tricky, right? So popping it into ChatGPT and saying, "Hey, can you just make this fit into 300 words?" You've put the time and effort into writing it and then ChatGPT just takes it that extra little bit to just get you across to the finish line.I feel like that's all right versus like, "Oh, Chat GPT, write me an entire story," and then you don't really have to think, research, whatever. It's kind of like this lost opportunity for learning because you're relying on it to do basically the whole thing for you and maybe you'll verify it, maybe not, I guess, depending on what kind of person is using the tool.JENNIFER: Yeah, I think that is one of the actual uses of these kinds of LLM tools that makes sense to have the author of a document use it to produce a summary that they can verify the correctness of or to do some style transfer to make it sound like a business email instead of something you wrote at 4:00pm when you're trying to just leave the office. But that's not really how it gets...That's not the limit of how people advocate that they be used. And it instead gets posed as, like, an ops tool. And people are proposing that you should do AIOps and you'll have machines that will just scale your systems for you or whatever or tell you that you're running out of database connections and jump right to an answer which may or may not be the correct one.And without letting any people go through the path of discovering what happened there. And if you're running out of connections because you have an N+1 query that is closing and opening new connections all the time, that's a very different problem than just like a runaway serverless system that is overloading your causing some sort of thundering herd problem for your database. And the AI probably can't tell you the difference. It probably doesn't know that there is a difference, but it sort of takes away the opportunity for people to do that investigation and learn what those things mean and what to do about it.ADRIANA: Yeah, that's a really good point. And I think to a certain extent, those of us who have not, quote unquote, grown up around ChatGPT, I feel like we could be...or similar tools for that matter...I feel like it's almost like because of how we grew up in tech, it provides like a guardrail for us where we're still probably more inclined to still do the research. Trust but verify, not take it for face value. But for folks who are coming up in this era of these various AI tools, I think it becomes a lot more difficult because this is like sort of they're not as encouraged to put in that extra thought or put in that bit of creativity before handing it off to the AI tool, because that's not what they've been brought up with. And I feel like that can be very dangerous then to the younger generation as a result.JENNIFER: Yeah, exactly. And so when you're like an experienced programmer and you're using ChatGPT to remind yourself what the syntax is for array map, that's a very different dynamic than inexperienced programmer who's using ChatGPT to create from whole cloth a function that will turn one array and do something to an array and give them different results.And now that...they don't know, superficially, these look like the same thing, but there's just so much experience and context that the senior person brings, and I think that we all forget that not everyone has that. And you had to develop it somehow, and it wasn't by having a computer just hand you an answer.ADRIANA: Yeah, exactly. It's those late night sessions with like StackOverflow open on different tabs, trying to figure out what is the right question that I should be asking and please let someone have had that same problem, so maybe I know what's going on.JENNIFER: For someone who hasn't even internalized what a callback is and how it works and how you should think about it, they're in such a different position, they can't even really do anything with the result other than just paste it into their editor and see if it works.ADRIANA: Yeah, it's basically like the blind programming at that point. I'm curious to see how things are going to pan out because there have been calls for legislation around blocking these AI tools to a certain extent, or limiting them. So I'm wondering how it's going to pan out. What are your thoughts?JENNIFER: I hope that something happens, because yes. I also have thoughts on the way that these tools get built and all of the harvesting of data without permission that goes into the training sets and how opaque they are and whether what the results are and how they get used and by who and on who. These are all big questions. So yes, I do think that, like, having some sort of, you know, government regulation oversight is going to be important because, you know, like, the way these AI models are built involves a lot of, like, harvesting of you know, people's work uncompensated from the internet. It's a very extractive thing. And then they get turned into these computer systems that people use to make decisions. And you can't really inspect those decisions. And people don't really understand what they're doing and how they work and then who uses them and to do what and who benefits and who suffers for that are sort of like...I don't want to say open questions, because the answer generally is going to be the same as it usually is. People with privilege will benefit from them and people with that will suffer. That's not great for us, but that is where things are going.ADRIANA: And what I find interesting about this whole thing, too, is that even some of the folks who are responsible for the creation of these technologies are sort of like, whoa, chill out. We gotta take a step back on this thing before it blows out of proportion, which I think is quite interesting.Now, I don't want to sound alarmist, but every time I see how advanced things start getting have continued getting with AI, I can't help but get Terminator vibes. I don't think it'll be quite so drastic, but I'm like, man, Skynet might not be too far away.JENNIFER: Maybe I think Skynet is the wrong thing to be concerned about, though. Well, I think it's important to note that the creators of all of these systems, what they're advocating for, is that other people should step back. They don't want governments to tell them to stop doing what they're doing. They just want governments to prevent other people from doing the same things and that's a different thing. And then you look at other AI systems that we have in the world, like self-driving cars. They get to a point where they can do simple things, fairly reliably in controlled settings, and then you unleash them on the real world and they're constantly going the wrong way down one way roads and, like, stopping for obstacles that don't exist and completely ignoring ones that do.I don't think that language models are going to be all that different. They don't have a real understanding of what's happening around them. They're just doing pattern matching and there's going to be patterns that they haven't encountered before. And what does it do in that case? Who knows?ADRIANA: And it's interesting that's for self driving cars, which can be scary enough if things go south, but then they're talking about...there's self-driving planes out there as well, which yeah, I feel like that's a whole other level of self-driving as well, which could be interesting.JENNIFER: Yeah. And like and it's easy to see how, like, the danger in like, self driving vehicles and why you would want to be careful about that. But then you turn it into language models and it's just like the algorithm that does things, but what you get is like self-approving mortgages and that's not going to be like, different. Yeah. Like that's still going to hurt people.ADRIANA: Yeah, absolutely. Wow. Damn. Yeah. The possibilities are endless, and it's not in a good way either. Cool. Well, I know we don't have too much time left, but I did also want to touch upon...you said when we were chatting earlier...you've got, like, a little project that you have that you've been working on around the Fediverse. I was wondering if you could tell us a little bit about that.JENNIFER: Yeah. So I have recently started a Fediverse project. So, like, I call it Letterbook Social. It's a Mastodon-like microblogging service. And the thing that makes it different, other than being just not Mastodon, is that I'm trying to optimize for the needs of the operators because it's in talking to people who run Mastodon servers now that I've met some of them in the last, I don't know, what is it, eight months now? That's actually not a great experience. Mastodon doesn't have very good Observability and it is hard to scale and it's hard to deploy and the admin and the operator tools and the moderator tools, for that matter, are very primitive. And so what I want to do is solve for that. I want to make it easy to set up and easy to scale and easy to understand what the system is doing and be able to oversee it as the human being running it, which I think is particularly important in this case because these things are very frequently overseen by one single person. Which is that can be very stressful, to say the least.ADRIANA: That is very cool. So are you building it on top of existing Mastodon code, or are you starting from scratch?JENNIFER: This is from scratch, since it seems like getting changes into Mastodon is sort of an uphill climb. And so I decided that since I don't particularly know or like Ruby anyway, that I'll just do my own thing. And so now I have a C# project. Get back to that lightning question, and I get to work with C#. And I've been doing a lot of that, and it's going to be a while before it's a usable thing. But it's getting to a point where in the near future I'll have something that stands up and operates and I can start exchanging messages with other services.ADRIANA: That is so cool. Really look forward to hearing more about that.JENNIFER: Yeah. Well, I'm sure I will be talking about it a lot on the Fediverse once I have something a little bit more concrete to talk about.ADRIANA: Very cool. Do you have anything on it right now, like any documentation or are you still so initial stages that it's like no, it's just the code.JENNIFER: Yeah. So I'm doing this like it's open source. I would be happy to have people help and contribute it's on GitHub. Like Letterbook is, I think, a pretty easy word to search for. And if you want a URL, there's letterbookhq.com.ADRIANA: Send me the URL and I'll include it in the show notes. Very cool.JENNIFER: I haven't had opportunity to focus on the kinds of open source project maturity that would make it easy for people to jump in and start contributing. But if somebody is feeling adventuresome, I would love to have more help, and I would be more than happy to talk through how things are structured and what contributions people can make.ADRIANA: Very awesome. So all you C# lovers out there, cool opportunity. Very cool. Well, we are coming up at time. Well, thank you so much, Jennifer, for joining today.JENNIFER: This was lots of fun.ADRIANA: Totally loved talking about AI stuff, your Observability endeavors, and your new little Fediverse project. Thank you so much for geeking out with me today. Don't forget to subscribe. Be sure to check out the show notes for additional resources and to connect with us and our guests on social media. Until next time...JENNIFER: Peace out, geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:Hazel Weakly (she/her/hers) spends her days working on building out teams of humans as well as the infrastructure, systems, automation, and tooling to make life better for others. She’s worked at a variety of companies, across a wide range of tech, and knows that the hardest problems to solve are the social ones. Hazel currently serves as a Director on the board of the Haskell Foundation and is fondly known as the Infrastructure Witch of Hachyderm (a popular Mastodon instance). She also created the official Haskell “setup” Github Action and helps maintain it. She enjoys traveling to speak at conferences and sharing what she’s learned with others.One of her favorite things is watching someone light up when they understand something for the first time, and a life goal of hers is to help as many people as possible experience that joy. She also loves swing dancing, both as a leader and a follower.Find our guest on:GitHubMastodonLinkedInDiscordRedditHazel's blogFind us on:All of our social channels are on bento.me/geekingoutAdriana’s TwitterAdriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:HaskellErlangTypescriptBlog post: So You Want to Hire for Developer ToolingSmallTalkObject-Oriented ProgrammingVisualBasic (classic)QBasicQBasic Nibbles gameQBasic Gorrila gameOpenSSLBleeding Heart (Heartbleed bug)Cost CenterAdditional Links:Video: Hacking the Pachyderm: Scaling Servers and PeopleVideo: OpenTelemetry Q&A Feat. Hazel WeaklyCatch Hazel at QCon 2023 in San FranciscoTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, Reliability, and everything in between.I'm your host, Adriana Villela, coming to you from Toronto, Canada. With me today, I have Hazel Weekly. Welcome, Hazel.HAZEL: Hey there. I'm glad to be here and I'm really looking forward to this episode. It's going to be a lot of fun.ADRIANA: Yay. Super excited. So first things first. Where are you calling from?HAZEL: So I am calling from the sunny surprisingly town of Redmond, Washington and if you were to ask me in a couple of weeks, I'm going to be closer to Seattle. Seattle. And we'll see how that goes. But yeah, I'm in Seattle.ADRIANA: Yay. Very exciting. Very exciting. You're my second West Coaster that I've spoken to for the podcast today, so I'm being outnumbered.HAZEL: I mean, West Coast is the best coast, and I'm sorry, I don't make the rules.ADRIANA: Fair enough. Fair enough. Yeah. There is something like, I think, kind of magical about the West Coast, where it's like, chill vibes at one with nature.It's a different vibe from East Coast, for sure.All right, cool. So I'm going to start with some lightning round questions.HAZEL: Awesome.ADRIANA: Prepare. Tun tun tun...Okay, first question, lefty or righty?HAZEL: I am a rightie.ADRIANA: All right.iPhone or Android?HAZEL: AndroidADRIANA: For personal use. Do you prefer Mac, Linux or Windows?HAZEL: Simultaneously, macOS and 'Nix OS.ADRIANA: All right, cool.Favorite programming language.HAZEL: Probably....I feel like I'm obligated to say Haskell because I'm on the board of directors of the Haskell Foundation and it's true that is one of my favorites.My cheeky answer is also that my favorite programming language is the one that I write in and nothing bad happens.ADRIANA: Awesome. I do like that.All right. Dev or Ops?HAZEL: YesADRIANA: Yes. Okay. Wow. Both. Okay.And final question. Do you prefer to consume content through video or blog post?HAZEL: Blog post. I read way too fast to sit there and not read it.ADRIANA: I am the same way. I was just telling my previous guest the same thing. She also prefers blog posts because I cannot sit there and just listen to someone go, "Blah, blah, blah," where I'm like, "Get to the point."HAZEL: I mean, developers being ADHD in this economy, who would have thoughtADRIANA: I know, right? For realsies. So I guess let's get into the meaty bits.Well, first things first. So before we get into what you do, tell us how you got to where you are. How did you get your start in tech?HAZEL: How I got my start in tech? That is a really interesting question because I have mostly had moments of a ridiculous amount of luck and then the ability to at the moment capitalize on that luck.So how I got my first job was I was at a programming lab at university and I ended up happening to overhear two people talking about Erlang. And I was like, "I know that language."Well, I didn't know it, but I knew of it, and what kind of weird ass nerd knows about Erlang in undergrad?So I ended up talking with them for 2 hours after the lab and one of them actually said, "Oh hey, I work at a company and soon we're going to have internships. Do you want to do an internship here?"And I was like, "Oh hell yeah, I do." Because I didn't have any other options.And so eight months later I actually ended up getting internship there after applying to bajillion other companies and none of them gave a shit about me because I hadn't even graduated yet.So it turns out that he was a racist, misogynistic, terrible person who liked to rant about like weird religious topics in the middle of a Costco food cart.But other than that, it was an interesting first experience.ADRIANA: Damn.HAZEL: But if it hadn't been for that one moment of me knowing about Erlang, I wouldn't have had that job and then I wouldn't have been able to seal up in all the random weird crap that I had to, that got me my second job.ADRIANA: And so what was so what was like? What was your second job?HAZEL: Oh, you want the full history?ADRIANA: I'm curious about the second job.HAZEL: So the first job I have finally had enough of the person being really toxic because he had actually gotten the other intern to quit.And I realized that his M.O. was going through and finding gullible college undergrad people. Getting them to be interns and then just having them be super cheap rate forever until they finally rage quit and left. And he never understood why everyone left and never talked to him again. Then I left and never talked to him again. Shocking.So the second job, I looked around and found a company that was hiring and they were hiring for a front end job in React and TypeScript and all that stuff. And I didn't know TypeScript at the time, so I took a brief six hours and learned TypeScript and then took the interview and aced the interview because I had actually ended up, out of purely coincidence, helping my father-in-law at the time get his website built for a construction contracting company.And because I had that whole thing built up from scratch, I had a huge amount of experience in that particular field that they needed.And they pulled up that website during the interview and said, "We want this."And I said, "Well, I know how to do it."So that's how that worked.And then during the first week there, I built out the entire backend in MongoDB, Node JS, TypeScript and did a whole bunch of ingestion from a very weird Microsoft Server database.That was problematic. The whole thing was problematic.It turns out that that company was a consulting company that was trying to use another company in order to bootstrap themselves without getting funding so that they could actually go and do the thing that they wanted to in the first place.So you had a whole company whose entire existence revolved around their one single clients, never figuring out why they were paying this much money for a single website.ADRIANA: Damm.HAZEL: I know.ADRIANA: Wow.HAZEL: Shockingly, that didn't turn out. I don't why.ADRIANA: What a shocker.HAZEL: So after about after about ten months of that shit, it all fell apart. But in that time, I became the senior-most engineer on the team, IC-wise, built out an entire component library, re-did all the local developer environments, rebuilt everything in Docker, did like, a 10x performance improvement on the entire website and a 10x performance improvement on load balancing.Got the entire back-end working more efficiently, leveled up the entire team in terms of being able to use the component library to redesign the entire website to meet the neurotic and weird ass requirements of a client that literally did not understand how things worked.ADRIANA: Damn. So was all this happening, like, while you were still a student?HAZEL: I, so I graduated halfway through my first job.ADRIANA: Okay.HAZEL: So I went from that first job to having designed the entire design system and done all those other things. That was about one to one and a half years after graduation.ADRIANA: Okay. Now, was area of study related to what your work was?HAZEL: My area of study was computer science and the university was Portland State University. It's a great university, but did it give me the tools that I needed to actually do the literal work? No, it gave me really good tools for understanding theory.ADRIANA: Okay.HAZEL: That didn't have a lot to do with programming. So, like my very first day on my job at my first company, I still remember it took me 4 hours to do some weird jQuery nonsense with a list in HTML.And finally the head developer person was like, "What's taking you so long?"And sat there and did the whole thing in front of me, essentially like ten minutes.And then he was like, dude, "I thought you were good." Basically.And then it was the weirdest thing was like his look on his face was, "I know you're smart, but what the fuck?"ADRIANA: Yeah, it's funny because I do find, like, school never quite prepares us for the workforce because yeah, I mean, it's too much theory, not enough practical stuff.The practical, quote unquote stuff that they make us do is so irrelevant that when you hit the workforce and you're solving the real problems of the real world, you're like, oh, shit, I have to learn this stuff from scratch.HAZEL: Speaking of things in the university world that are not relevant at all, but I had a lot of fun doing. One of my favorite things that I ever did was during the operating systems course, we had to take a toy kernel and implement a multilevel feedback process, scheduler and priority queues.So I implemented that with the vast majority of the state machine logic being implemented in about 80 to 100 lines of C preprocessor macros that were recursively, expanding using macros and a whole bunch of various extraordinary crimes.My code was beautiful. It magically scoped in variables that were hidden. It did a whole bunch of things. It relied on some undefined behavior. I had to turn on GCC pragmas so that things actually compiled because Dead Code branch elimination wasn't working with ternaries and it was glorious.Absolutely none of the TAs after the third assignment would touch my...like, none of the graders would touch my code. The only person who would grade my code was the TA who was a grad student and she's still a friend to this day. But that code was cursed.ADRIANA: That's awesome. That's awesome.HAZEL: It prepared me for TypeScript, is what I'm saying.ADRIANA: So it's funny how the little things prepare us for the things that we don't even know are in store.HAZEL: Right?ADRIANA: Yeah. I feel like my whole life has been that. So one of the things that we have chatted about previously is, well, you've got many hot takes, and I feel like...we've talked about your thoughts and feelings around platform engineering.So I was wondering if you could share that with folks because platform engineering is the hot topic of the day and everyone's got an opinion. There are tons of discussions going around. So throw your thoughts into the ring here.HAZEL: Wow. Did you allocate the full four days required for this podcast to have all of my thoughts on platform engineering?ADRIANA: No, shortly, no. I mean, sadly, noHAZEL: Sure. So I'm going to have to summarize a bit. I wrote a blog post recently called "So You Want to Hire for Developer Tooling?" And in there I talk a lot about the first platform engineer or the first people that will become platform engineers in your company and how to not fuck it up.And I'm sure a lot of people are going to read it and then fuck it up anyway, and that's fine. It's really hard to hire for it.But hot take-wise, platform engineering is something that I find really interesting in that in the industry I see this habit of over and over and over.Someone says, "Oh hey, in a sociotechnical system we need to solve the technical problems for the spice of the social problems and solve the social problems for the spice of technical problems and have them work together in a collaborative fashion, understanding the constraints and challenges of both."And then someone goes in and says, "But tooling and vendors" and the whole thing goes to shit.And this repeats over and over and over as organizations don't want to skill up in the cultural maturity and outsource understanding of something that they see as not a core driver to the business, which in modern economic theory, it makes sense.If it's not a core competency of the company, why should you not externalize it and view it as a cost center?And since the world is being eaten by tech, people haven't seemed to catch on to the fact that developer tooling, how developers work, the entire process of how they collaborate with each other and with the company is now inherently a core competency of existing in a tech-driven world.So if you want to be relevant as a company, it's not platform engineering, it's not DevOps, it's not tooling...you need to understand how people work together and how people and solutions and technology work together and how to scale that understanding.And the problem you will always run into when doing that is if your work is meaningless or if you are ruled by toxic work behaviors or you have a bunch of institutional biases and corruption in your company that prevent people from genuinely being able to improve the system as they see it, you will always end up with a broken system.And so if you talk to executive leadership and say, "Oh," and they ask you, "What can we do to improve developer productivity?"It's not productivity you want to focus on; it's the developer experience. And the developer experience there. The biggest leverage you're always going to have has nothing to do with the tooling, has nothing to do with Kubernetes, has nothing to do with fucking YAML.Although swearing has a lot to do with the YAML because it's a natural and necessary defense mechanism when you have to write it every day.With that aside, if you're going to improve the experience of developers at a company, the work has to be meaningful, the work has to be high impact. The work has to be high leverage and the relationship that the company has with the developers needs to be healthy and fulfilling and equitable.And you will find very little leadership that is willing to take the full implications of that and execute on it.ADRIANA: Yeah, I completely agree and I think that's, I mean, that's why we see so many of these so- called "transformations" fail miserably. Because leadership doesn't have their heart into it.HAZEL: Mmhmm.ADRIANA: They've been told by someone else who's like, "Hey, do this, it's in vogue."HAZEL: Yeah.ADRIANA: Go on, go forth, do it. And it's just like business as usual.HAZEL: Did you know that related to that, it turns out one of the best indicators of quality in the system is whether or not people genuinely enjoy working on it.And if you ask the question, "How does this system, how does your experience with interacting with this make you feel?"Does it make you feel more alive and more whole?If the answer is "Yes," it's probably a good quality system.And if you need to choose between what to do, you can always ask yourself, "Which of these options will make me feel more alive and whole when interacting with the system?"And a lot of people will go, "But what about the quantification?"Like, what about the numbers? This seems like hippie mumbo jumbo.And no, you should be in touch with your fucking feelings. You should be in touch with the human side of yourself, and you should not just bury it deep in your ass crack in the name of capitalism.It's literally more efficient to actually think about your feelings and think about what it means to be a human and the human experience and try and make the world a more wholesome and inclusive place for everyone around you.It's literally more efficient. It's literally objectively superior in many ways. And you can show that.ADRIANA: Yeah, that makes a lot of sense, actually, because I've definitely felt in the past, like, when I'm working on something where I feel like it's a pleasure to come into work every day because this work is fucking cool. I think you're going to see it in my code.You're definitely going to see that extra. I'll go that extra little bit. Right? Just to get it done. Because I'm excited about what I'm doing. Because I think this is cool shit that we're trying to do here.HAZEL: And you're invested in improving a system that you care about and that, you know, cares about you.ADRIANA: Exactly.HAZEL: No one wants to work on a broken tool and no one really wants to fit a broken tool if they don't think that it will actually be received. Like, you can't fit something that won't be integrated into the system and it cannot see improved something that doesn't want to be improved.ADRIANA: Yes. And it's both by design, doesn't want to be improved by design and doesn't want to be improved by the human overlords of that tool. Right? Because I've been in so many situations where you come in and they're like, "Oh, yeah, this thing's a piece of crap and it could really do with refactor or rewrite." But then no one wants to invest the time. Right?I've felt so many times where it's like this thing just straight up needs to be gutted. Like, keep somebody keep a team working on this thing to keep the lights on while the other team does the rewrite, right? So we can all be happy in the end, but a lot of organizations aren't willing to invest that extra time and money, right? Because that means you've got an overlap of like, two teams working basically on the same thing.But I feel it ends up being a very short-sighted decision to not support those types of things because you're shooting yourself in the foot in the end.HAZEL: Yeah. And so with migrations in general, one thing that's really interesting about that is it turns out there's like a pretty formulaic strategy you can use in order to execute a migration of arbitrary size and complexity. So there's three main steps that go into a migration.The first step is de-risking a migration. So that involves talking to people and understanding what they actually need and working with the people that are being hit the most by the inefficiencies and the insufficiencies of the current situation. And then you get them the new situation and you make sure that it works. You work with them, you work on that. You make sure that this thing will actually do what you want it to. That's the first step.The second step is the enablement, where you say, "Okay, what is all of the low-hanging fruit?""What is all of the automation we can do?""How can we take this migration?"And as much as possible, take the toil out of it and take it out of the hands of people who don't have the context required to execute the migration. How can we facilitate that?And the third step?The third step is literally someone needs to sit there and A commit to finishing it and B commit to communicating about it.So the thing that I find fascinating about migrations is that most migrations fail in the first stage of knowing, actually sat down and talked to the team before they ripped out a solution. Like people will rip out a solution that isn't broken or people will try and say, "Here's a new solution" that actually makes the problem worse.If you just talked to people and actually worked with them to verify that something will be in fact the solution, you would save millions of dollars a year or hundreds of millions or even billions in your company over time. And you would save years of developer effort by just fucking sitting down and talking. And it's ridiculous that this is not a thing.The second stage migration.ADRIANA: Yeah, I actually agree with you.HAZEL: And the second place the migrations fail the most is people celebrate the automation step and the majority and they don't celebrate the done done of, "We actually finished everything and turned off the old system."Don't celebrate if you haven't turned off the old system and sit there and commit to fucking doing the last mile. If you initiate the migration, it's on you to finish it. You cannot just hand that off to someone. It's on you to finish it, but it's also on you to communicate about it.And so many migrations are not able to be finished because the communication of your progress, the communication of the value, and the communication of what needs to happen in order to actually do this, never happened. So again, talking to people, or rather the lack of talking to people, kills most migrations. And it's astounding to me because sure, it's difficult to understand what leadership or what your management or other stakeholders or other teams are looking for in understanding the progress of your migration.But it turns out there is a simple and effective strategy to figuring out what they need in order to feel like you're communicating with them.You ask, "Hey, is this working for you?"And they say, "Yes," or "No."And if they say, "No," you change it and then repeat, right?ADRIANA: That's it novel concept, right?HAZEL: It's not like we've had language as a society for like 18,000 years or something, right?ADRIANA: I know, right? Yeah. But it's so true and I think that's like the most fundamental problem that we see across the board with these types of initiatives.My favorite example is always, like, infosec. I worked at a bank a gajillion years ago and we were like so we had admin mode. Like, developers had admin mode on their laptops and we were able to install certain software so that we could get the job done.And this was, I believe, it was like, pre- approved software to begin with, but then all of a sudden, InfoSec, one day they're like, "By the way, we're going to block the installation of all software unless it's on a whitelist."And then unfortunately, we had to discover as we went. Like, "Oh, shit, this is blocked."OK, well, now we need to contact InfoSec to whitelist this. And we couldn't complete the simplest tasks. I mean, it was ridiculous. All of a sudden, developer productivity went to a standstill because InfoSec didn't bother to speak with development teams to talk about, like, "Hey, what would your workflow look like if we did this? Right?So it was just like the directive came from whomever. And thou shalt live with this heinous crime against development. So yeah.HAZEL: I mean, the real solution there, the real solution there obviously was to have all the developers move to Visual Basic and Microsoft Excel as their main development platform. Because it turns out Microsoft Excel is one of the most efficient, beautiful, and glorious development platforms out there. And it's one of the best programming languages, too.ADRIANA: Oh interesting, Excel. Yeah, I guess so. Yeah.HAZEL: Honestly, it is actually a good programming language because it turns out so one of the people that says that is Simon P. Jones, who is one of the people that helped create Haskell.So actually the Visual Basic inside Excel is a functional programming language that has first-class functions. It has a whole bunch of other like, not-so-nice things in it. It even has lambda functions. It has all the fun, hot, trendy things. And the reactive programming model inside Excel was later stolen and turned into ReactJS.I'm modifying the history. I'm going to pretend it was stolen from Excel. But Excel is actually pretty awesome. Like, in terms of a programming language, it would be hard to find something that is more accessible to people outside of tech.ADRIANA: Yeah, that's true. You're talking about, like, straight up Excel formulas at this point. Are you talking about, like, okay, not the fancy stuff that you can do with, VBA?HAZEL: They're the same thing. You can stick VBA inside basically any Excel cell.ADRIANA: Very true. You know, like speaking of VBA and Visual Basic. I liked Visual Basic. That was like, I guess, officially my second programming language. I started on QBasic back in the day. Yeah, back in the day when it with the gorillas throwing banana bombs to try to destroy the city. And there's also the Nibbles game. Big fan.HAZEL: Nice. So you've always been a BASIC bitch is what you're saying?ADRIANA: I mean, BASIC was basic, but I liked Visual Basic. I thought it was intuitive. It was a nice way to develop GUIs I mean, especially when you go from Visual Basic and then try to develop any GUI stuff in Java. That was a fucking nightmare. And I did not last more than 2 seconds trying to sort that out before going, "Buh bye!"Yeah, Visual Basic was great. It was fun. I used it in high school. Like, my high school programming class was Visual Basic and built some cool stuff. I did some shitty animations with Visual Basic. It was great.HAZEL: Visual Basic is so underrated and so related to Visual Basic. A lot of the programming languages and environments of the previous decades got an incredible amount of things very right. And so one of the things that I've actually said decently often about DevOps and infrastructure-as-code and all these things, is it's really just people trying to recapture the fever dream hyper productivity of SmallTalk and the SmallTalk VM, but with an audit trail for compliance.That's it. Trying to manage a wibbly wobbly ball of state in real time, at scale, without fucking it up. But the best feedback loop and the best productivity you have basically ever really been able to get in terms of being able to dig into a system.If you've ever seen SmallTalk, honestly, it's incredible.ADRIANA: I've never seen it. I've only heard of it my dad used to code in SmallTalk back in the day. Would not shut up about it. Yeah, that was because I think it was like one of the original object oriented languages out there. Right?HAZEL: It was THE object-oriented programming languageADRIANA: There you go. Yeah.HAZEL: And every other language after SmallTalk took object-oriented programming and said, "What if we ruined it?" And then proceeded to do exactly that?ADRIANA: So basically no one has succeeded in capturing the glory of SmallTalk, is what you're saying.HAZEL: And we probably never will, because now people are really used to being able to undo something or statically analyze something. And honestly, both of those are extraordinary inventions like the ability to say, oh shit, never mind, is actually really good. However, that has a lot of false confidences, in that, in the real world, your system is actually pretty mutable and pretty ugly anyway. So for people to say, "Oh, we can just undo this," or "Our state doesn't actually REALLY exist," that is kind of untrue. And so pretending that that's the case leads to developers having this very mismatched and distant view of the system that they work with.Whereas in SmallTalk, if you fucked up production, you knew the second you hit enter, because you just crashed the entire VM and the entire company is now screaming down to its knees, sobbing, the whole thing fucked up. But you KNEW...INSTANTLY.ADRIANA: Right. Yeah. So you get that immediate feedback versus the pussyfooting around maybe there's a thing that's wrong.HAZEL: Yeah. And the immediate feedback, it makes you fear yourself the appropriate amount. Like, if you release code that's about to nuke production, "What could nuke production?" You're gonna double- check it, whereas now, we're just like, "It's stateless. It's in Kubernetes. It's totally fine. We can undo this."And then you actually delete half your database and then OpenSSL Bleeding Heart happens and then all these other things happen, and it turns out that you're just like, you're crying in a corner, you eat 20 years in five days, you're like, stress bleeding out your toes. It's a whole thing.ADRIANA: That's that's actually a really interesting way of viewing it because yeah, I agree. It's similar...This reminds me of the argument where making developers responsible for their code once it goes into production, rather than throwing it over the wall, right? Because if you're the developer responsible for your code, there's no fucking way you're going to let shitty code go into production, because you're the one who's going to be on the hook if something happens.HAZEL: Right. A lot of people will think about that and they'll go, "So if I just make everything the developer's responsibility, it's all better." And that's not true. Because, with equal power comes equal responsibility.But with equal responsibility needs to come...With greater responsibility needs to come greater agency. Agency and responsibility cannot be separated because they are the same thing. And if you pretend that they're not the same thing, you're going to end up with a bunch of pissed off, burnt-out developers who hate you, your whole company, everything about you, and they're going to burn the entire economy down to the ground.ADRIANA: Yeah, absolutely. I yeah, and I think, unfortunately, that's kind of how I felt early in my career. I was just so fucking burnt out. Like my first job out of school, I was pulling these ridiculously long hours where I was working six days a week, 14-hour days.And I remember I complained...this other guy, and I complained...like, I was fresh out of school. This other dude, Dale, he was engaged, so he's like, planning a wedding, and we're both like, "What the fuck, man? This is like, way too much work. We're dying."Like, we have no life. And so we're like, okay, we're going to complain together, right? And then Dale bailed on me and I complained to my boss and then he's like, "Oh, you can have the weekend off."And I felt so guilty. I felt so guilty for taking the weekend off because the rest of my team was like working and Dale bailed on me. So I was like the little prissy-ass bitch who was complaining about, "Oh, she can't handle the work."But as a result of that, I had zero vested interest in seeing that thing succeed because I was like, I hated that system. I'm like, if it goes down in flames, I do not care because they treat me so badly. I don't care. I don't care about my work.HAZEL: And if you were to take any of the executives and just talk to them and say, hey, you just ruined any capability that this company had of building a team that is engaged and able to actually put everything where it needs to go, they would just look confused and go, well, this is a cost center, so why do I care? But the knowledge required to operate and build and improve this is really about something that fundamentally can't be a cost center.ADRIANA: Yeah, and that's interesting because I feel like this whole, like, "You're a cost center" argument ends up really interfering with innovation and productivity because, well, it costs too much. We can't invest in that. You're not making us money. And it's to the detriment of the entire organization as a result.HAZEL: Yeah. One of the things that I always try to do when I am leading infrastructure teams is I see infrastructure as the way, or not "The Way," but as "A Way" to enable people to have low risk, high quality, rapid experimentation.You want that experimentation to be risk-free, and you want to basically sow the seeds of innovation. And the way to do that is you get a bunch of people that are smart, you put them in a room, you more or less let them do whatever the fuck they want to, as long as they have like, a vague sort of agenda that they're kind of going towards. And then you let them try out as many ideas as possible, and you let them understand the system.Because this whole idea of software development, or development, or building a platform is really about, "How do I understand this so deeply and intimately that I can express the entire understanding of this problem in a way that other people can interface with this as if it was knowledge made concrete and tangible, and interactable."And that requires you to try out a whole bunch of things that are not that thing. It requires you to evolve the understanding of that thing over time and the understanding of the knowledge itself, the process of getting that knowledge, and the process of even thinking about what it means to communicate about it.And that's what you're doing. It's not programming. It's knowledge work. It's creation of understanding itself.ADRIANA: I think that's such a cool approach, because I think by having these loosely-defined borders...parameters...it opens up your mind to creativity.Because now it's like, oh, I feel like if you let people do their thing, I think they will naturally gravitate towards finding the problems to solve and then they will be excited about solving those problems. And like you said, they'll learn things along the way. And for me, I think one of the coolest things after solving a ridiculous problem is taking a step back and thinking, holy shit, look at all the things that I learned along the way to be able to get here and having there's no better way to inject enthusiasm into a team than doing that.Personally, I always tell my bosses, "I don't like being bossed around." I thrive...And that's the thing I appreciate about my current boss is...They know that I thrive from doing my thing and doing it well, and finding cool problems to solve and then writing about it or whatever. Like sharing the knowledge in whatever way.And I think more managers need to recognize that because the field that we are in is ultimately a very creative field, contrary to popular belief.HAZEL: It's one of the most creative fields out there. And one thing that I think of, that you reminded me of is we have the concrete work of doing something, and then we have glue work, which is tying together things in a way that is not necessarily recognized. But there's a third secret option. It's not glue work, and it's not the concrete things. I'm going to call it innovation work. It is work of finding inspiration and drying it out and bringing it to life and sowing those seeds.And it's not glue work. It's not concrete work. It is the work of divining inspiration itself from sources around you and making that visible and making the process visible and figuring out what it means to be innovative and to execute on visions you don't even know you need to look at.ADRIANA: It's yeah, and sometimes that means like, finding collaborations in the periphery of what you're doing, or finding connections to your work from somewhere that you wouldn't necessarily see that connection, because everything I think, brings us to where we need to be.It's kind of like what you were saying. I think we were talking about this earlier. Career-wise. All the things that we do, all of our experience leads us to where we are now. And you draw on that experience. You draw also like what you said on the serendipity and the opportunity taking advantage of lucky situations. I mean, you're only truly lucky if you take advantage of that situation.And I think a lot of us tend to not recognize when we are in a lucky situation and that like, hey, this is something that I need to grab a hold on before it goes away.HAZEL: Yeah. And fine-tuning that notion of luck and that gut instinct of I should focus on this or I should prioritize. This is something that I've done a lot of and it's been one of my greatest career accelerators. Because that fuzzy feeling of this is important, or this person is cool, or this is where I need to be in right now, or I need to go into this room. I don't even know why sometimes, but I just trust it because it's going to lead me to pretty cool places like here.ADRIANA: Yeah, actually that's a really, really excellent point is trust your gut. Trust the fuzzy feelings.HAZEL: Unless it's talked about, then don't touch it.ADRIANA: I was just going to say we are coming up on time. But before we go, I would love to get any hot takes or words of wisdom for our viewers and listeners.HAZEL: So a hot take. Someone asked me once to explain what Kubernetes was because they felt like it might have been a practical joke or something, because they're trying to figure out what it is, and there's just so much nonsense going on.And so my explanation of what Kubernetes was...is...Kubernetes is what happens when you take about five to ten years of institutionalized tech debt, reinvent it, and create an entirely new parallel universe of tech debt as a consequence. Which is to say that it is highly effective, and yet much of it, to some degree or another, is simultaneously needed, yet unnecessary.ADRIANA: That is awesome. I think that is my favorite view of Kubernetes to date. So thank you. Awesome. Well, thank you so much, Hazel, for joining me today on Geeking Out. Y'all. Don't forget to subscribe and be sure to check out the show notes for additional resources and to connect with us and our guests on social media. Until next time.HAZEL: Peace out and geek outADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet.Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:Reese Lee (she/her) is a Senior Developer Relations Engineer at New Relic, where she enables users on open source technologies such as OpenTelemetry. She has spoken on various topics related to OpenTelemetry, and maintains and creates community resources aimed at OTel end users. She is super into anything paranormal, and enjoys sci-fi and traveling.Find our guest on:TwitterMastodonLinkedInFind us on:All of our social channels are on bento.me/geekingoutAdriana’s TwitterAdriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:Meme - Don't wait until you feel confident enough to actReese' Etsy ShopOpenTelemetryOpenTelemetry on YouTubeOpenTelemetry End User Working GroupOpenTelemetry Q&A sessionsOpenTelemetry in PracticeOpenTelemetry in Practice Meetup GroupOpenTelemetry Monthly DiscussionsOpenTelemetry Feedback SurveyBlog post: Tail Sampling with OpenTelemetry and New RelicVideo: Why, How to, and Issues: Tail-Based Sampling in the OpenTelemetry Collector - Reese Lee, New RelicVideo: OTel Me About Metrics: A Metrics 101 Crash CourseAdditional Links:Adriana's talks on YouTubeTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, Reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada.I've got with me Reese Lee from New Relic. Yay!REESE: Hello. I'm so excited.ADRIANA: I am super stoked to have you on because we work together on the OpenTelemetry End User Working Group. So we're like, always chatting.REESE: Yes. Huge supporter of Adriana and so honored to be here. So hopefully we have something good for y'all.ADRIANA: Yay, super excited. And Reese, where are you calling from today?REESE: I am calling from Vancouver, Washington. So not Vancouver, BC. Lovely Pacific Northwest. It's very nice out right now, so that's why I'm in the sun.ADRIANA: So awesome. Yeah, I had to retreat indoors. It is exceedingly humid here in Toronto. Yeah, we had a bit of a reprieve from the excessive heat that's been going around in most places, but the last couple of days have been sweltering. But I don't think I'm any better indoors because my office is super hot, but I...REESE: Oh, no. Yeah.ADRIANA: But I can't record outside because it's so noisy. There's always cars driving by and I always sit on my front porch when I work outside. So, alas, I can live vicariously through you .REESE: Oh, no. Yes, for now. It's funny, it's been a pretty mild summer. We've even had a couple overcast rainy days. Usually by now we'll have seen some or more 90 degree days than we've seen so far this summer, so we'll see.ADRIANA: Fingers crossed.REESE: I can't believe it's August already. I guess, yeah. We still have more hot days ahead.ADRIANA: Sorry?REESE: Oh, just saying we still have more hot days ahead. It's not August yet.ADRIANA: Oh, yeah, I know, right? Yeah, I think it's coming for us.REESE: Yeah.ADRIANA: Okay, so I'm going to start off before we get into the regular format. Well, I guess this is part of the regular format. I want to do some lightning round questions. I promise they'll be fun. Don't panic. You are the first person I'm subjecting to these lightning round questions.REESE: Let me do some stretches.ADRIANA: You have the distinct honour of being the first person I'm recording, so I'm testing out my format on you. Okay, question number one. Are you a lefty or a righty?REESE: Righty.ADRIANA: Are you iPhone or Android?REESE: AndroidADRIANA: Mac, Linux or Windows? What's your preference? Not what you work with, but what you would prefer to work with.REESE: Mac.ADRIANA: All right, favorite programming language?REESE: .NETADRIANA: Cool.REESE:ADRIANA: Dev or ops?REESE: Ops? I don't know.ADRIANA: And there's no wrong answer. And final question: Do you prefer to consume content through video or through blog posts?REESE: Blog posts.ADRIANA: I'm the same way. When someone throws a video at me, I'm like, Nah.REESE: Yeah. I like to be able to read it and see the words. And sometimes people just either talk too fast or they talk too slow.ADRIANA: Exactly. It's like sometimes I can read faster than they can speak, so let's get on with it. And then you can skim.REESE: Yes. And then depending, sometimes they have weird...they try to do funny little things, which is fine, but sometimes I'm like, I just need the info. I don't have time for 20 seconds of you being silly.ADRIANA: Yeah, it's true. I know. I keep thinking that, too. Like, whenever I read blog posts. And sometimes I have a lot of preamble in my blog posts, so then I feel guilty for doing it.But I'm like, I think somebody somewhere enjoys this. But for those who don't, I have clearly marked headings so that you can skim to the goodies.REESE: Yes. It reminds me of recipes where you just want the recipe and they go into the whole backstory of how this recipe been in the family for generations, and they like to make it on rainy Sundays or whatever. And I'm like, oh, my God, I just want it.ADRIANA: Yes, it yeah, I'm totally with you on that one. Like, on these recipe sites, I'm like, Just give me the damn recipe.REESE: I'm already hungry. I don't have time to...ADRIANA: I know, cut to the chase and can I have shortcuts?REESE: Yes. What do I need to buy from the grocery store?ADRIANA: Well, there you go. You survived my lightning round of questions, so let us get into the meaty bits. All right, so I guess let's start with the beginning. Like, how did you get into tech?REESE: Yeah, I've really only been in...I'm still a baby, I think, in the tech industry. Before I came to New Relic, I had my own small business, which I did for several years.I was in recruiting for a while. I was in retail management for some time. And after several years of running my own business, I was ready for new challenges, and I was also ready for a steady paycheck again.And I enrolled in a local coding bootcamp, actually, which, incidentally, is where a lot of Relics also come from. Got into tech support at New Relic, which is what I did for the first couple of years and then kind of happened upon Developer Relations. And so I've been doing that for almost a couple of years as well now. I'm now a Senior DevRel Engineer with a specific focus on open source technologies, specifically OpenTelemetry, which is how Adriana and I met. And yeah, I am here now.ADRIANA: Wow, that's so cool. That is, like, quite a varied path.REESE: Yeah, it has been. I kind of wish I'd checked out tech sooner. I always had this idea that programming was very dry, very boring. That was just a perception that I had. And I had a friend, she's an artist. She used to make apparel. She has some really cool stuff. I actually have quite a few of her pieces, and she went to the coding bootcamp, and I was like, what really? That interests you? And she was the catalyst, really, for me, looking into software programming.And that was how I learned that it's actually really creative and it's all about creating, just using a different format from like, I used to make jewelry and art. And so I love creating stuff, being creative.And when I learned that, holy shit, programming is actually a very creative activity, just using a different medium, I was really intrigued, enrolled literally, like, two weeks later in the same bootcamp that she did.ADRIANA: Yeah.REESE: And yeah, haven't looked back since.ADRIANA: That is so awesome. It's funny because I always have said that programming is a very artistic endeavour. I think you met my daughter the one time, and she is so artsy, and both my husband and I are in tech. And she's like, "Fuck no, I do not want to do this. I do not want to sit at a computer all day long." She's an artistic person. She wants to be a dentist. So I think she prefers to sit at a dentist chair hovered over somebody's face than at a desk. But hey, each their own.But I keep telling her, I'm like, "Hannah, honestly, you'd be such a kick-ass developer." I mean, not that I'm not going to force her either way. She takes her own path. I am fully for that. But I always thought that if she allowed herself, she would actually be a very kick-ass developer.REESE: And she might and she might eventually take her own path to yeah, like, if I'd known sooner, I definitely would have checked it out way earlier than I did.ADRIANA: Yeah.REESE: You just never know.ADRIANA: I feel you. I started out early in programming, but then I rejected it for a really long time. When I went off to university, I'm like, I'm never going to touch coding again. But I studied engineering, and it's part of the curriculum. You have to code as part of it. But I think when I started studying in school is when I finally had these "Aha!" moments because I was mostly a self- taught programmer.I'd get those big thick books from the bookstore and learn QBASIC. That was my first programming language for me, that was like, oh, I don't want to do any of this stuff. This seems horrible. I don't have the brain for this. And then I discovered, oh, I actually do have the brain for this. I was just, like, looking at it completely differently.REESE: Yes, absolutely. Along with thinking it was dry and boring, I also thought, oh, I don't have my brain. Doesn't work quite that way. I'm still learning a lot. There's still a lot of technical things to learn to practice.But I didn't even know at one point that it would be a thing I would be interested in. So I really hope people, especially those that are kind of sitting on the fence or they also think it's boring, I hope they somehow find this and they're like, oh, maybe I should check it out.ADRIANA: Yeah, let them be inspired!REESE: Mmhmm.ADRIANA: So your business before you went into, like, you took the coding bootcamp, so you were selling, like, jewelry. You said you're making your own.REESE: Yeah. So there's really two small businesses that I was running. One is Chubby animal illustrations. It was mostly like, fat cats. And I also had brass and gemstone jewelry that I would make. So I did a lot of local regional art shows, also had my products in a lot of stores over the country and had international customers. And it was really fun. And I still do that on the side a little bit now to kind of keep that juice flowing. Creative juice flowing is what I meant.ADRIANA: That's awesome.REESE: Yeah.ADRIANA: I have to say, running a business is so hard because I did that for a year. And after a year, I was like, no, I'm done.REESE: What did you do?ADRIANA: I decided to quit tech in like, I don't know, it was like 2013 or 2012 to become a professional photographer, and it was going to be like, "Bye bitches, I'm done." And I came back to tech a year later.REESE: I mean, to be fair, I do know professional photographers as well who work for themselves, and it does take a lot to build up to that place.ADRIANA: Yeah, exactly.REESE: And years, for sure.ADRIANA: Yeah.REESE: Yeah.ADRIANA: That was the thing. For me, after a year, I was at a crossroads. I'm like, if I love this enough, I can keep at it and maybe I'll even build my business. And I was starting to see I had made some changes to how I approach things, and I was starting to see the changes, but by then I was like, this is exhausting. I don't actually want to do this for a living. I actually loved tinkering with doing my website and I liked blogging and sending out my newsletters, which is a lot of DevRel-y stuff. And I'm like, oh, I had more fun troubleshooting the PHP and some plugin that I bought for my WordPress site than having to deal with the day-to-day BS of running a business.And then I'm like, you know what, it's time for re-evaluation. And I came back to the exact same team. And for me, I think the hardest thing was getting over the fact of what will people think of me? Because I was like, "Bye, I'm done." And then I'm like, you know what, I don't care what people think. I'm doing this for me, not for other people's perception. So screw it.REESE: Yes. That is so important. That's such an important thing that I hope we really impart to everyone who's listening is that do what you got to do. Don't worry so much about what other people are thinking.ADRIANA: Yeah.REESE: They're going to think what they're going to think. No matter what you do. If you think you're doing the right thing, other people could still be judging you and thinking you're not doing the right thing. So who cares? Just do what's right for you. Try out what's good for you and go with that. People are going to think what they're going to think anyways, and just let them be. Just focus on being the best thing for yourself.ADRIANA: Yeah, totally. You do you. So what made you sort of decide, like, you wanted to put a pause on the side...I guess they weren't side businesses at the time, the actual businesses to shift over to do this bootcamp. I know you said you were inspired by your friend who...artist friend who was doing this bootcamp, but what kind of made you sort of take a step back and think, maybe I need to change.REESE: Well, kind of like to what you said, running your own business is hard. There's definitely a lot of great perks. So my work schedule was almost whenever I wanted. I would just obviously have to make sure for the shows that I wanted to do, have those scheduled and make time to make the products for those shows. And the landscape has changed so much too since I was last really in it with social media. A lot of people run their businesses online fully as artists, and that can still be really hard because of the algorithm. And like, oh, now Instagram wants you to do reels. And you don't really want to do reels. You just want to take photos of your products, which, as a consumer, I prefer sometimes to just look at photos because not everyone's good at doing videos. And if I have to rewatch a video four times just to see the products, that is so annoying.So the landscape has changed a lot and of course the pandemic. I think things are kind of getting back to normal. But I went to the bootcamp in 2019, which is a year before everything shut down, timing-wise it was really interesting because all of a sudden the markets were like, can we be open? Should we not be open? But people need to depend on this for the livelihoods, we don't know what to do.And so I viewed it as a very serendipitous time for me to have made that change because by that time everything shut down. I was already at New Relic doing tech support and seeing my artist friends kind of like trying to figure all that out. I felt very grateful. And to answer your question, yeah, I have been doing it for several years. I really enjoyed it. But it is definitely a huge grind. If you're not working, you're not making money a lot of the time, like there's online sales and stuff, but you still have to promote, promote, promote, make sure your products get seen it's a lot. And paying 300 something dollars a month for health insurance was starting to get real tiresome.ADRIANA: Oh, yeah, yeah, I totally agree.REESE: Yeah. So I was also thinking I just wanted to try something new, look for some new challenges and get into something I've never done before. And so I also viewed my friend's trajectory as very opportune because it led directly to my timing of my switch and everything world events happening. So, very grateful.And for anyone who's listening, if you're thinking about trying a new career, do it.Just try it out if you can and see if you like it or not.ADRIANA: I mean, if you're in a position where you're like, I can afford to check this out for like a year and see how I feel, absolutely, go for it. Because there's always going to be an excuse, right, to not try.REESE: Yes, I think a lot of people think, oh, I got to do this and this and they have this laundry list of things to do before they're ready and...I saw this meme that I think will illustrate this point more and I'll send to you so you can put in your show notes or something, but it was something like, confidence doesn't come from being ready. You have to take action to build that confidence, if that makes sense.ADRIANA: Yeah, it makes sense. I think it's like one of those gradual things, right, that builds up. Like...REESE: So, like, yes, okay, "Don't wait until you feel confident enough to act. Your confidence builds as you take action."ADRIANA: Yeah, that's perfect. I love it.REESE: That was the thing. Yes, I messed it up, but that was it.ADRIANA: That's awesome. Words to live by.REESE: Yeah, I screenshot it because I was like, I'm going to Oh, cool. share this with people and I will send that to you're...ADRIANA: Yeah, I want to put that in the show notes. That's awesome. Cool. Now I'm beholden to including these in the show notes. Dun dun dun. Cool.So you mentioned your journey into tech and then so you started in tech support once you got your first tech role, and then what brought you into DevRel and then specifically, like OpenTelemetry?REESE: Oh, man. I really enjoy tech support, and there's aspects of it that I miss even like, solving, just like spending time trying to figure out these technical problems that our users are having. I really enjoy that piece of it. I do still get to do that a little bit in my current role.So I've been doing tech support for a couple of years. I knew I wanted eventually to explore beyond tech support and most tech companies, there's so many different paths you can check out, right?Software engineering, product management, technical training. There's so many other paths also that I was not even aware of, like program management, talent acquisition.And so I knew I wanted to try other things. I kind of decided on one route specifically. And so I was just really open at the time to just seeing what was available and looking for opportunities to add to my skill set here at New Relic. I didn't know anything about Developer Relations. Did not even know it was a thing.There were two engineers that I had worked with in tech support. They worked on the .NET Agent, and both of them reached out to me separately about this new role that was on their team. They were on the OpenTelemetry team. And at first when the first engineer reached out to me, I took a look and I was like, oh, that's very cute. That's very nice that you thought of me, but I don't know about this. And then when the other engineer reached out to me, I was like, okay, maybe I should believe in myself a little bit more.ADRIANA: Yeah.REESE: And so I ended up reaching out to the hiring manager, and she and I chatted more about the role. And that's when I found out about Developer Relations a little bit more. And the more I learned about it, the more intrigued I became because it's a lot of different things rolled into one, which is very great for my ADHD brain.I need a lot of different things to keep me stimulated and motivated. And yeah, after chatting with her more, I decided to go through with the internal application process and I was hired on.And that was when I really dove into OpenTelemetry and learned more about what it is, exactly what it means for Observability moving forward, and landed in the OpenTelemetry End User Working Group, which is how I met Adriana.And yeah, it's been a wild ride. I've since published several blog posts. I've spoken in front of a live audience of hundreds of people, which I used to be extremely, extremely shy. Like, even in college, I had trouble speaking in front of the class and would need my professor to help me because I was extremely shy. And so the fact that I have now spoken on technical topics in front of hundreds of people is amazing to me.ADRIANA: And you're a great speakerk. I can vouch for that. I remember attending two of your talks and I'm like, damn.REESE: Thank you. Oh, my God, that means so much coming from you because I don't know if you all have seen Adriana's Talks, but she is awesome. Her slides are amazing.ADRIANA: Thank you.REESE: Yeah. So thank you.ADRIANA: Awwww...And hopefully we can do a collab talk at some point. Fingers crossed. That our KubeCon. Or is it Observability Day? We applied for one of those together. Observability Day, I think. Fingers crossed!REESE: Those notifications, I think, come out next month, I think so? In, like, a week...ADRIANA: I'm going to be on edge. Yeah. Because I think August 6 is the deadline for the CFPs for KubeCon and Observability Day.You're right. You're right. Yeah. The deadline.ADRIANA: I guess they'll let us know in a month.REESE: Yes, you're right. Okay.ADRIANA: We'll see. We'll see. So...And we applied also for DevOps days, Montreal. Let's see how that goes. Fingers crossed.REESE: Yes, I know. Oh, my gosh. I sort of got...every week is, like, such a whirlwind.ADRIANA: I know, right? CFPs are so exhausting to do and I guess for our listeners who aren't familiar with the term CFP is call for proposal. Right? Usually that's what it stands for.REESE: Yeah.ADRIANA: So it's like when there's a conference or whatever, they put out a CFP, so you have to fill out an application for speaking. Like, it's basically a proposal for a talk. And if they like you, then you get to talk. But it's a lot of work to put together a CFP, and every conference has its own nuance, so then you have to tweak it for their specific ask, and then they'll limit the number of characters for a certain description. So then you write this beautiful thing and it's like, I'm sorry, you're over by 100 characters. You're like, fuck.REESE: Although I have found usually for me, it's easier to have more and then subtract versus, like, oh, I might not have to add more.ADRIANA: Yeah...REESE: But yes, it's a whole project in itself, like, submitting proposals and then trying to come...ADRIANA: And then when you get rejected, it's like, "They don't love me. What's wrong?"REESE: Trying to come up with topics that I think are timely, relevant, interesting, not just to the audience, but something that I might want to learn, too. Oh, man.ADRIANA: Yeah, exactly.REESE: It's a whole thing.ADRIANA: Yeah, because that's part of the, I guess, CFP game, if you will, is like, you don't necessarily have to be an expert in that topic, but if it's something that you want to learn more about, it's an opportunity. Like, if you get accepted, it's like, well, I guess I'm digging into this now.REESE: Yes. And so actually, my old manager, she pointed out something. Well, I guess I kind of done it because I submitted a talk proposal about tail sampling in The Collector within, like, a month of me starting my DevRel job and not really knowing too much about OTel at the time, and found out a couple of months later I got accepted, and I was like, oh, shit, I really got to learn up and down.But she explicitly said she would submit talk proposals on topics that she wanted to learn about that she didn't really know. And I was like, that is what I did without knowing it. And it's such a great idea to make sure you have time to learn something that is your job and that you can now teach to other people.ADRIANA: Exactly. Exactly. And, you know, I keep running into your tail-based sampling article every so often, and it's very well-written. Like, Reese is a very good writer. So good speaker, good writer. You got the whole package.REESE: Thank you. I do have a degree in magazine journalism, so I would hope I am a good writer.ADRIANA: That is so awesome. Oh, my God. You're like Mystery Woman. So many skills.REESE: Finally putting my degree to good useADRIANA: That's so cool. I did not know that. Damn. This is why we have podcasts, so that we could learn more about each other.REESE: I really enjoyed writing, and when it came time to apply to university, I was like, writing journalism. But then with me being so shy as I was at the time, I just found it really hard to be in a reporter mode.ADRIANA: Yeah, because you have to be outgoing.REESE: Once I start talking to people, that's fine. I've gotten some good interviews in the past, but going out, I'm much better at it now. And I know a lot of people will be shocked to learn that I was so shy, but I was. And I've definitely had to get out of the comfort zone and talk to people on my own.ADRIANA: Especially when you're running your own business, like, you got to promote yourself. That's what I found scary about running my own business, where I'm like, oh, God. Because I'm the same as you. I am mortified of talking to most people. When I was a kid, even going up to an information desk at a mall was like it took me half an hour of convincing myself, okay, I can talk to this person and forget talking on the phone. That is a mega phobia of mine. But to run your own business, you have to put all that aside and just suck it up and do it. And I do feel like for me personally, it helped a lot, and it sounds like for you as well.REESE: 1000%. Oh, you know, other things that have helped along the way too, being in retail management and just people feeling like they can just come up and say whatever they want to you.ADRIANA: Oh, jeez.REESE: And I worked in recruiting for a while for an agency, and so I did a lot of cold calls.ADRIANA: Oh, jeez. That's very...REESE: You have to get over that real fast as well. Yeah, it's been interesting, like post college, just been putting myself in these situations where I'm like, okay, I have to talk to people.ADRIANA: Well it paid off.REESE: Yes. Now speaking live in front of ike 500 people.ADRIANA: Yeah. Like, KubeCon EU, you had quite the audience for your Metrics talk...REESE: YesADRIANA: And it was a good talk.REESE: thank you.ADRIANA: You're welcome.REESE: YesADRIANA: So we're almost at time, but I did want to spend a little bit of time talking about the OTel end user working group. I figure we should always promote the fun things that we do. So yeah. Why don't I let you explain what it is and the cool things that we do?REESE: So the OpenTelemetry community, I'll just take a step back a little bit. So the OpenTelemetry community, like, I think a lot of open source communities, is made up of many special interest groups or SIGs, as well as working groups or WGs, but I guess we can just say working groups that all have specific purposes to develop the community and the technology that it's part of, right?And so the End User Working Group, it was actually created by my former manager of the OpenTelemetry team here at New Relic. She saw a need to create a space where end users could come together and have access to resources to help them learn about OTel, adopt OTel, implement OTel, as well as a space for vendors and other interested parties who want to help develop OpenTelemetry to create resources for end users.And so to that end, we now have several monthly events that we do. So you'll see Adriana and myself at a lot of these.We have the End User Discussion Group, which is where end users come together and discuss challenges that they face with their migration. We also have a guest maintainer or someone from the technical or governance committee come on to help provide additional insight to the project and help answer some of the deeper technical questions that we may not be as aware of.We also have OpenTelemetry in practice, which is kind of an hour long, or they're about presentations about OpenTelemetry. So, like, for instance, we did one recently about how Observability is a team sport which is about adopting OpenTelemetry at a specific company. It was Farfetch with that one. That was a really good one.We had someone come on and talk about what distributed tracing is and how to do it with OpenTelemetry.So that's two.We have another one, which is the End User Q&A/interview/feedback session. It's kind of all those things. And that's where we sit and chat with an organization who is adopting or has adopted and implemented OTel in their organizations, and we find out, why did you decide to migrate to OTel? What are the challenges that you face? Like, how did you migrate? Because a lot of end users are interested in that. We also do blog posts which from some of these events that you can see on the OTel blog.We also have a community survey that anyone is welcome to take and share. Whether you are just kind of starting out in OTel or you have already implemented OTel in production, we are always looking for ways to improve the project. And that was one thing I forgot to mention at the beginning, which is one of the goals of the End User Working Group.So besides being a space for end users and developers of OTel to come together, we also want to create and maintain a constant feedback loop from end users to maintainers with the ultimate goal of improving and advancing the project.So these are the activities that we started and are doing pretty much every month to meet those goals. And so that's why feedback is so important to us. Connecting with end users is so important to us. And if you have an OTel story to tell, we would love to hear it. We know other end users would love to hear it. And so I know Adriana will get all these in her show notes about how you get in touch with us and stuff. So I will get that to her and yeah, I feel like there's other stuff that I might be forgetting but those are the main ones.ADRIANA: I feel like you got the important ones yeah. I do feel...like you said, we do blog post summaries of some of these events, and then we also put up some of the videos for these because that way we cater to our video lovers and our blog lovers alike. Which I find that they're usually very well-received. Like, whenever I post them on socials, people really seem to enjoy the content.So it makes me really happy that even if you're not able to actually attend the session, you can still benefit from it after the fact. And I always find in these sessions, you always learn new things, especially end user discussions where because we do usually have somebody from the OTel community who's in attendance and they'll mention stuff that being worked on where you're like, oh my God, I didn't know that was a thing. Even so, I always find it's super useful to attend these and there's always something new to learn and then there's people with really gnarly ass use cases for OTel, sharing some of the questions that are asked. I'm like, oh my God, this is awesome. I did not know about this.REESE: Yeah, and so if you're interested and you don't feel like you have specific questions but you kind of just kind of want to listen in, please feel free to do so. There's no obligation for you to ask questions if you don't really know if you just kind of want to sit in and learn and don't even know what questions you might want to ask yet.ADRIANA: Yeah, so true. It's such a great learning opportunity. I know people always will tell me after the fact, too, that they always get so much out of these sessions. If you're an OpenTelemetry fan, highly recommend.REESE: Yes, and then you can come chat with Adriana and myself real time.ADRIANA: And we've got our third co-lead as well, Rynn, who also...they did a lot of the OTel in Practice in the past as well, and a lot of organizing. Like there's a meetup group for OTel End User Working Group, so I know they're always keeping that up-to-date.Tons and tons of stuff that we work on behind the scenes and there's three of us running this thing and there's so much work to do. But it's good that there's stuff keeping us busy because it means that there's demand from the community.REESE: Yes. And it's also a great know if you want to contribute to the community but not necessarily with code contributions. We could always use help with content creation. Oh, there is one more thing. The YouTube channel. The OTel YouTube channel.ADRIANA: Yeah, that's right. Yeah, that's where we post our videos after. Yeah, we edit them so that they're not like boring.REESE: Adriana does a fabulous job with that. But yes, we are going to try and do more with the YouTube channel as well with creating content.ADRIANA: Yes, content creation, especially video, is like so much work. Oh my God.REESE: Yep, yep.ADRIANA: Like even a short little Instagram reel, you're like, Holy crap. Especially when you're like fiddling around on your phone going, how do I use this little tiny thing here?REESE: Oh yes. Yeah, going back know, running like my own small business, product photos, product descriptions, all of those things, all of those things take so much.ADRIANA: Time and effort. But we do it because we love it.REEDE:Yes.ADRIANA: Cool. Well, as we wrap up, I wanted to know if there are any parting thoughts that you would like to share, any inspiration with our audience.REESE: Whoo. I feel like I did I did so much better in the lightning round that than now, I guess since we kind of talked more about the career side of things. One thing that I try to impart on my colleagues to help them with their own career growth is especially as so much of us work on remote teams.Now, do your best to be visible with your communications and your contributions online so that your teammates, your future managers, future teammates see and shout out your co-workers.Talk about the problems that you've solved. Obviously not in a braggadocious manner, but just be like, hey, I did this thing, and I want to thank so and so for their help. It goes such a long way for people to see that who may not necessarily work with you a lot over Zoom or in person.And that has really helped me with my career growth here, is being very visible in my online communications. And yeah, from that, I've had people reach out to me about internal opportunities. So I really want to encourage people to be as visible as you can.ADRIANA: I think that's really, really great advice. And being visible means sometimes it means tooting your own horn and it feels weird, but it's got to be done, otherwise nobody knows what you've done.REESE: Yes, it's a skill that I am still learning as well. But think about, like, when other people do it, you're like, oh, my gosh, I didn't know you did that. That's great. Great job.ADRIANA: It yeah, totally.REESE: So think of it from that perspective. And it's rare that I'll be like, look at this person bragging about themselves.ADRIANA: Yeah.REESE: I never really thought that at all when I've seen something like that on Slack. And so if that's your concern, don't let it.ADRIANA: Yeah, I totally agree. And the same goes especially for an outward-facing role like ours, doing it on social media. People are not going to see your content unless you put it out there and it feels weird. I admit sometimes some of the stuff I post feels weird, but no one's going to know that I've done this unless I post it them.REESE: Exactly. I mean, you know, no one knew what Apple was at one point. They had to get the word out there.ADRIANA: Exactly. And I also really liked your other point of giving shout outs? Give credit where credit is due. Because I think we don't get here just from climbing the mountain solo. We've had a lot of help along the way and making sure that folks get that recognition, because I do feel like what goes around comes around. There's good karma to be had. Paying it forward, I think, goes a really long way.REESE: Yes, 100%.ADRIANA: Cool. Well, thank you, Reese, for joining me here today. This was a super awesome conversation. And thanks for letting me guinea pig you on my lightning round questions.REESE: I am so honoured, and thank you so much. I'm so excited to see what you do with this new podcast.ADRIANA: Yay. I'm psyched. Thanks for following me on my journey. Well, everyone, thank you for joining today on Geeking out. Don't forget to subscribe and be sure to check out the show notes for additional resources and to connect with us and our guests on social media. Until next time, I am your host, Adriana Villela, with Reese Lee, signing off with...TOGETHER: Peace out, geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About Hannah Maxwell:Hannah is a high school student living in Toronto, Canada. She’s also the daughter of OCMM co-host Adriana Villela. She loves showing off her artistic side, whether it’s through baking, sewing, painting, or knitting. You can see her creations on Instagram. Hannah has been bouldering since she was 3, and still enjoys going to the bouldering gym with her parents. Although both of her parents work in tech, Hannah would rather not spend her time sitting at a desk and staring at a computer all day. That’s why she’s planning on becoming a dentist when she grows up. Find our guest on:InstagramFind us on:All of our social channels are on bento.me/geekingoutAdriana’s TwitterAdriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyTranscript:ADRIANA: Is this thing on?HANNAH: Is this thing on?ADRIANA: Is this thing on?HANNAH: Is this thing on?ADRIANA: Ready?HANNAH: YeahADRIANA: Hey Y'all. Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, reliability, Observability,TOGETHER: And everything in between.HANNAH: I've heard this too many times.ADRIANA: I'm your host, Adriana Villela, and with me for our very first episode of Geeking Out, I have my wonderful daughter and producer and just know overall, all around wonderful person.HANNAH: I was going to say overall, just know your emotional support human. Actually, I think I cause you the opposite.ADRIANA: My emotional support human, Hannah, and this is the first episode of Geeking Out, and I'm super excited to be bringing you this episode. You know, if you were a fan of the On Call Me Maybe podcast, you know, it's sad that the podcast has come to a close, but we have started this podcast talking about similar subject matter to On Call Me Maybe. And this time it's been a treat because Hannah is helping me with production of this podcast, because she has mad video editing skills.HANNAH:I wouldn't say mad.ADRIANA: And as a teenager is very tapped into this Youtube sub-culture...HANNAH: I know the social media. I'm very self conscious of people walking by...ADRIANA: Because we're recording outside. This was Hannah's idea. It's kind of nice. I mean, look, it's the end of summer. We're enjoying some outdoor time, right?HANNAH: Yeah.ADRIANA: Cool. Well, okay, so this very first episode of Geeking Out, I'm going to start off with a series of lightning round questions that I usually ask all my guests, and I guess my first guest is Hannah, so I'm going to read them off my phone. Sorry.HANNAH: She doesn't have good memory.ADRIANA: I don't.HANNAH: Sorry if there's noise, but hopefully the microphone...ADRIANA: I really hope so. Okay, first question. Are you a lefty or a righty?HANNAH: Righty. But I wish I was a lefty.ADRIANA: Because all cool people are lefties.HANNAH: Like, honestly, leftiness is cool.ADRIANA: It IS cool. I have to say, I love it. Okay, next question. iPhone or Android?HANNAH: iPhone.ADRIANA: Mac, Linux or Windows?HANNAH: Mac? Is that correct? Is that correct?ADRIANA: There's no wrong answer.HANNAH: No, but I don't know. I only understand Mac.ADRIANA: Okay, Mac it is. Okay. Favorite programming language.HANNAH: HTML or Excel?ADRIANA: I think Excel kind of counts.HANNAH: Yeah.ADRIANA: You'll see more in a future episode with Hazel Weakly.HANNAH: Yeah, but in my business class, we had to basically code inside the cells of Excel, and then we had to make an invoice for a magic store.ADRIANA: Yes, true. Okay, next question.HANNAH: And HTML is great. I love HTML.ADRIANA: I don't love HTML, sorry. Okay. Dev or opsHANNAH: Dops.ADRIANA: Dops! [laughts] JSON or YAML?HANNAH: YAML! It sounds like, "ya", "yak", and "llama"ADRIANA: Okay, awesome. That's great reasoning behind it. What is your superpower?HANNAH: Okay, so I thought about this long and hard. Long and hard. Long and hard. Okay, that's actually a good question.ADRIANA: I know. I just stuck it in, actually. This is a new. New for...HANNAH: Okay. Super speed, because I feel like you can do so much with that.ADRIANA: No, the question is not what superpower would you like? What is your actual current superpower? I'm super glad to know, though, that you would love super speed. I think that's actually a really good one, because it'd be nice to do.HANNAH: I'm so embarrassed.ADRIANA: Don't be embarrassed. This is hilarious. Don't worry. Wait 'til you watch one of our future episodes where I basically invent DevOps all over again. Ha. Ha. Ha. So if you feel embarrassed, don't worry.HANNAH: Okay. That felt like a job interview. Like, what's your superpower?ADRIANA: Yeah. One of the questions that I actually like to ask when I interview people.HANNAH: My superpower is at certain times, I can be very laser focused on what I'm doing and get it done really effectively. And I care so much about my work that I'm going to make sure it's the highest quality that it can be. I'm never going to slack off on anything if I truly love it.ADRIANA: That's true. Yeah. I can definitely vouch for that, which is why Hannah is such an awesome producer, video editor, social media, manager of this.HANNAH: Did I get the job?ADRIANA: Yes. Congratulations. Congratulations. You got the job. Okay, final question, and this is actually our topic for today, which is, do you prefer to consume information through text or video?HANNAH: Video definitely. I cannot...like, things just don't process as much if I read. I can read an entire book and be like, I kind of know what it's about. But then if I see it visually and I have someone kind of telling me the parts with actions and videos, I understand it way more. Like for a Shakespeare test, I cannot read very well. It does not process in my head.So it was the night before the Shakespeare test. I'm like, "Oh, my God, I don't understand any of this."ADRIANA: I mean, Shakespeare requires a secret decoder ring to begin with.HANNAH: Yeah, but I'm like, "I do not understand this. I'm going to fail this test." And so I watched YouTube videos breaking down each scene, and I'm like, "Oh, that makes sense."ADRIANA: So then you pass the Shakespeare test.HANNAH: Yes.ADRIANA: And then some.HANNAH: Yeah.ADRIANA: And you know what? I think that's actually a really important thing to discuss that a lot of people don't discuss, because I think there is a lot of emphasis in reading when kids are growing up in school, et cetera. And I think yes, I think we all need to learn how to read.It is an essential skill.HANNAH: But you don't need to be able to read high level books.ADRIANA: Yeah, I feel there's a lot of people are looked down upon if they're not amazing, avid readers. I do love reading. I do love to get lost in a novel, but Hannah doesn't. But I don't think any less of her or I don't think she's any less intelligent because she doesn't like to get lost in a good book. But you like to get lost in a good YouTube video.HANNAH: Yeah, people get so mad at someone deciding to watch a movie instead of read. They're both entertainment. They both teach you things. If you have the same story in a book and a movie, they will end up teaching you the same things.Who cares if it's on a screen or in a play or on a book? Like with text, it really doesn't matter as long as you end up getting the same or at least good things out of it.ADRIANA: Yeah, so true. Yeah, it's funny because even as you're growing up, we watch a lot of TV. No harm in that.HANNAH: No harm in that.ADRIANA: No harm in that.HANNAH:I don't like the stigma around like...ADRIANA: Oh, you watch so much TV, you must be a horrible person.HANNAH: I've learned so much from even TV shows like "Friends".ADRIANA: So many good life lessons.HANNAH: Or "That '70s Show" or "My Little Pony".ADRIANA: "My Little Pony" is a legit good show. Honestly, every adult should watch. It';s got such wholesome messages. I love it.HANNAH: Yeah, but it just really bothers me that people think I'm less of a human because I use YouTube videos or TV shows to learn when they're basically the same thing as a book.ADRIANA: I don't, by the way. I honestly...I embrace your learning style.HANNAH: I know, but when I tell someone about the Shakespeare story, they're like, "Well, that's cheating."ADRIANA: Not cheating.HANNAH: And it's like, but I bet if you're in my place and you just can't process stuff with reading...I can go through an entire page and not know what it was about and it's a struggle with me and I just feel like I should play to my own strength.And there's nothing wrong with playing to your own strength. What would be cheating was if you had the YouTube video playing on a headphone while you're doing the test. That's cheating.ADRIANA: That is cheating. Yeah, I totally agree.HANNAH: It's definitely not cheating to just learn your own way. And there's nothing wrong with however you learn as long as you end up learning from it.Like, maybe if I was just watching all these videos or TV and I'm not learning anything good from it, maybe then you'd have to be like, well...BUT, I'm learning something and I don't think people should get upset or tell you that you're learning wrong.ADRIANA: Yeah. Yeah, I agree. And I think it's really important because everybody has different learning styles. I'm not a video person, so usually when someone sends me a YouTube link, I'm like, this better be a short video and it better have captions because that's going to capture my attention.But if I need to learn something, I much prefer skimming through a blog post.But I appreciate that that does not work for everyone and different strokes for different folks. And I think we shouldn't look down upon anyone because they learn in a different manner. I think what I find really awesome about you is you've found out what works for you in terms of learning and so you basically hacked yourself. Right?The Hannah hack.HANNAH: But also I think it's more important now as we're trying as a society to destigmatize learning disabilities and different ways that people learn.I think one of the things that we do need to look at is we can't get mad at someone for learning in a way that maybe you don't learn.Because I just get really annoyed when I say that I don't read books and people look at me like...ADRIANA: "Oh, you're an idiot."HANNAH: Yeah. It's actually very upsetting because I spend a lot of time doing the things that I like and learning about them, and I have end up learning about them and I'm proud of that.But then for someone to say, "Oh no, you didn't learn it the right way" It's like, but I still ended up learning.ADRIANA: Hannah wants to become a dentist when she grows up. And guess what? You learned a bunch of dentist facts. So oh, tell us some dentist facts.HANNAH: Charcoal is bad for your teeth. Do not brush more than like two times a day. Also, this is not professional advice. This is stuff that I've learned and that braces are sometimes on dogs.You don't want to mess with braces stuff.ADRIANA: You don't want to put braces on dogs? Or you do?HANNAH: No, you do want to put braces on dogs. If you have braces, you don't want to do anything else that the Orthodontist doesn't recommend, like trying to find a hack to move faster.ADRIANA: Oh, yeah, fair enough.HANNAH: I get that social media is like, yes, there's a lot of bad stuff that comes with it, but youalso have to think of it as a different perspective on someone who has actually been helped by social media.I don't think I'd be the same person that I am if I didn't have social media.ADRIANA: That's true. Yeah. You've learned so many cool things, like the most random stuff, like Hannah's been getting financial advice from YouTube.HANNAH: I have! You know, all about investing.Yeah, but I just think that we really need to stop thinking of social media as a bad thing.Like this horribly toxic thing.ADRIANA: Like you said, there's bad aspects to it that are soul sucking and trolls are horrible and that'll mess with your mind.But we also need to remember that social media can be a really cool source of education and I think we need to embrace that as well.HANNAH: Yeah. And someone said to me, well, you're just "mindlessly" scrolling on YouTube? And I'm like, but then if I hadn't mindlessly been scrolling, I never would have actually found these things. I don't think I would have discovered different trying to think of something I've learned, like stuff about job interviews or disability rights issues if I hadn't been just like,I don't look for these things, but because they came to me, I'm interested in them and I can spread my information that I've learned.And so there's nothing wrong with scrolling on social media because it'll take you to places that you never would have gone otherwise.Because I don't think I would have actively been looking for financial advice. Yeah, financial advice, something like that. So I just think it's a really helpful tool. Yes.I'm not saying that there isn't bad stuff. There's a lot of bad stuff and misinformation, but you have to look at the light of and think of someone else's viewpoint of it.And we can't just say get rid of all social media, because then that would be getting rid of a resource.It would be like getting rid of a library.ADRIANA: Yeah. Because it's basically demonizing the whole thing, where it's like, well, guess what? There are bad books out there.HANNAH: Yeah, they're books with, like...ADRIANA: Misinformation...HANNAH:...comments...Exactly. Yeah, there is misinformation books.ADRIANA: So how's that any different? It's just, back in the day, we had books. Nowadays we have books and videos, and others...we've got, like, social media. So it's just like the evolution of how we consume information, which I think is super cool.And I think really the moral of the story is don't judge somebody by how much they read or don't read. Judge them by like...HANNAH: what they've come out of it with.ADRIANA: Yeah, exactly. And being tolerant of how people consume information and understand that not everybody's going to learn like you do.And I think it's been really cool getting to see like, you know, I learn in a very particular way and you learn in a very different way. And you also like to learn by sometimes by watching others being shown. Which is something that her teacher in the early years of Montessori, her teacher Cecile, figured out early on.I mean, that's so cool to be able to understand how we learn so that we can be our better selves, right? Because there's no way that we can improve if we can't learn.And part of hacking ourselves is understanding how we consume information.HANNAH: Some may say, oh, well, you're getting better at spelling and vocabulary when you read. You can still do that by watching something. Yeah, I can learn new words by someone just saying them. And I can be like, "Hey, let me look it up.""What's that word?"Or if I like to watch stuff with subtitles on, I'm like, "Oh, that's how that's spelled."ADRIANA: Yeah, that's so true. There are just so many ways that people can learn, and the more we say, well, this is bad, you're just trying to stop people from learning, which.You're excluding an entire group of people who choose not to learn or cannot learn in the same manner.HANNAH: Yeah. So I think that's all I wanted to say, you know. I may come back for another episode about this. We might have a guest on who has an opposite opinion.ADRIANA: Oh, yes, on social media. Yes. Hannah's friend Alice we've tapped her to come for a future episode. We're very excited to have her on.Yeah, well thank you Hannah for joining me today.I hope y'all enjoy our inaugural episode of Geeking Out.Make sure that you follow us on all the socials...HANNAH: And let us know what you think about this topic because it's a very interesting topic that a lot of people have a lot of different opinions on, so I'd like to hear those different opinions.ADRIANA: Yeah, definitely.And the interesting thing about this topic too is it's not a technical topic, but it applies to pretty much any job out there, right?Whether or not you have a techie job like me or a non techie job like you want to pursue.I think understanding how we learn makes us better at doing the jobs that we want to.Yay.Well thank you, Hannah.And with that, thank you so much for joining us on Geeking Out.Be sure to follow us on all all the socials. We've got show notes coming your way.Until next time...HANNAH: Peace out, and geek out.ADRIANA: Geeking Out is produced and hosted by me, Adriana Villela. I also composed and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally designed all of the cool graphics. Be sure to follow us on all the socials, by going to bento.me/geekingout.