Geeking Out with Adriana Villela

About our guest:Juraci Paixão Kröhling is a seasoned software engineer, a Governance Committee member for the OpenTelemetry project, and an emeritus maintainer of the Jaeger project. With a strong focus on observability and open-source development, Juraci has delivered talks at conferences such as KubeCon EU, KubeCon NA, OpenSource Summit, Devoxx Belgium, FOSDEM, and various DevOpsDays. With deep expertise in distributed tracing and observability, Juraci empowers software engineers to optimize their applications and build reliable observability pipelines. Currently working at Grafana Labs, Juraci continues to shape the future of observability tools while passionately contributing to the open-source software engineering community. Outside of work, Juraci is a proud parent of three kids and finds solace in the hobby of sleeping, albeit occasionally interrupted by the delightful chaos of parenting responsibilities.Find our guest on:LinkedInX (Twitter)GitHubFind us on:All of our social channels are on bento.me/geekingoutAll of Adriana's social channels are on bento.me/adrianamvillelaShow Links:Love ParadeDomain Specific Language (DSL)Julius Volz, PromLabs FounderJulius Volz Prometheus videoOpenTelemetry Collector SIG on CNCF Slack (special interest group)OpenTelemetry Collector Contrib on GitHubCanadian National Exhibition (CNE)OpenTelemetry Contribfest at KubeConOpenCensusOpenTracingRedHat OpenShiftOpenTelemetry CollectorReal User Monitoring (RUM)OpenTelemetry Protocol (OTLP)statsdOTel Logs Bridge APISLF4J (Java Logging)slog (Go Logging)OTel End User Working GroupOpenMetricsOutreachyAdditional Links:Dose de Telemetria - Juraci's weekly Portuguese language livestream on OTelRecap of Adriana and Ana's talk on Platform Engineering at KubeCon 2023Adriana’s Observability Day talk on the Observability of CI/CD Pipelines with co-speaker Reese LeeTranscript:ADRIANA: Hey y'all, welcome to geeking out. The podcast about all geeky aspects of software delivery DevOps, observability, reliability and everything in between. I'm your host Adriana Villela. Coming to you from Toronto, Canada and geeking out with me today is Jurassi. Welcome Judasi.JURACI: Thank you very much. And I'm surprised always when speaking English, people have a trouble speaking my name. That's not the case with you. You were perfect.ADRIANA: Oh, thank you! So, Juraci, where are you calling from today?JURACI: I'm calling from Berlin, Germany. Yeah, I'm freshly moved from Brazil back to Germany to Berlin. I'm here since 9 August, so I'm here less than a month now.ADRIANA: Very cool. I've been to Berlin once when I was working in Munich, I took a train to the Love Parade.JURACI: That's nice, that's wonderful.ADRIANA: This was back in 2000.JURACI: Okay, yeah, that's nice.ADRIANA: I'd never been to the Love Parade. I was like, wow, it is an experience. It was a fun experience. So that's my experience with Berlin. I hope someday to actually see Berlin properly.JURACI: Well, almost nothing here can top Love Parade. So you've experienced Berlin on its best.ADRIANA: Awesome. So before we get going with the main content, I have a few lightning round questions that I like to ask all my guests. So don't worry, they're painless and they're fun. So let's get started. So first question, are you left handed or right handed?JURACI: Oh, I'm left handed.ADRIANA: Me too. I'm so excited to meet left handed people! Yay. Left handed and Brazilian. Best combo ever. I'm slightly biased. Next question. IPhone or Android?JURACI: Oh, Android, that's easy. Open source? Well not so much, but yeah.ADRIANA: Cool. Next one, do you prefer Mac, Linux or Windows?JURACI: That's also easy. Linux.ADRIANA: Awesome. Hardcore. Die hard, very cool. Okay, favorite programming language?JURACI: Oh that's tough. I don't know, I'm using Go most of the time now, but Ruby still has a place in my heart. But I was a Java developer for so long, so I don't know, I mean a mix of Java, Ruby and Go now.ADRIANA: Nice. Most of my career was as a Java developer. Sixteen years. So I have a love hate relationship with Java.JURACI: I guess that's why I like Ruby so much, because Java provides you the security in so many aspects and Ruby is just like this nice language that is beautiful to read and it's fun to write. It might not be a perfect fit for everything, but it is a fresh view of the programming world for a Java programmer.ADRIANA: Yes, very true, very true. Actually I know a lot of people who love Ruby because I think they describe it as being like a very simple, very elegant language.JURACI: It is. Not only the language itself is very nice, but what you can do with that, you can build very beautiful DSLs with Ruby. And they really feel like DSLs, like domain specific languages. And when you build a DSL in Java, for instance, it still feels like Java. Right. But there are some DSLs in Ruby that you cannot tell it is Ruby. You really think that it's a new language built on purpose for that specific domain? I think that's what makes Ruby beautiful.ADRIANA: That's very cool. Next question. Prefer Dev or ops?JURACI: Dev.ADRIANA: All right.JURACI: I've been operating my own servers since like ever. And I love doing that. I ran my mail server for more than a decade now. I stopped a couple of years ago. I decided not to continue doing that for my own sanity.ADRIANA: That's a lot of work.JURACI: It is, but that side of operations is really very close to my heart. But still, I'm a developer.ADRIANA: I feel you. All right, next one. JSON or YAML?JURACI: Oh no. None?ADRIANA: That's a fair answer.JURACI: Well, if I have to pick one, then YAML, of course. But yeah, no...I don't know.ADRIANA: Yeah, I prefer YAML over JSon. JSON's too garbly for me. Too much happening. It gives me Java vibes like so many curly braces.JURACI: Yeah, and double quotes everywhere. In YAML you can just choose when to place it and when not to place it. And comments, I mean, you can place comments on YAML, you cannot with JSON.ADRIANA: Oh yeah, that's true. Score for YAML. Okay, next question is, do you prefer to consume content through video or text?JURACI: Okay, I'm still a text consumer, so books, articles. But I am on this verge of consuming more and more media in podcasts or presentations, like recorded presentations from conferences and so on. I find that. I think the best balance right now is a mix of all of them. There are some great content that has been provided in forms of tutorials and presentations at Kubecons for instance.ADRIANA: Yeah.JURACI: That you cannot find written anywhere, so you have to go and watch them at the same time. I find good old books very pleasant to read.ADRIANA: Yeah, I definitely have to agree with you. Text is my default. I love a good podcast, especially like when I'm doing, running errands, walking, doing housework. It just gives my brain something to do. But yeah, I agree with you that there are some cases now where video is the only way to consume the content. So you kind of have to just power through.JURACI: Had some, I don't know, I had some like, oh, YouTube. YouTubers. No, I refuse to do that. But then I think we are past that now. People are producing great. I mean, look at Prometheus, right? So Julius Volz is creating great videos on Prometheus and there is no one better, or there is almost no one better in the world that can talk about Prometheus. No bigger authority than Prometheus and him than Julius. And it's only in video.JURACI: Of course he does have his courses and some written content, but the videos are just great.ADRIANA: That's awesome. Good to know. Good to know. And final question in our lightning round, what is your superpower?JURACI: Oh, getting my kids in bed. I can do that better than anyone else. I just get them there and they fell asleep in a few minutes.ADRIANA: Yeah, that is very impressive. I just have the one daughter and when she was little, oh my God, the excuses, the excuses.JURACI: And I'm leveling up. I take care of two now because my youngest one is too young for me, I cannot breastfeed her. So mom still has to get her to sleep. But we are now transitioning also to no breastfeeding anymore. So in the future it's not going to be two kids, but three to get in bed.ADRIANA: So then you'll really put your superpower to work. Amazing. Well, awesome. Thanks for playing along with the lightning round questions. So now onto the good stuff. So OpenTelemetry...I want to talk to you about...because you are one of the OpenTelemetry maintainers, right? What's your specific role in the OpenTelemetry community?JURACI: I wear quite a few hats actually, but two of them are really big. And perhaps the biggest hat that I have right now in the OpenTelemetry community is on the Collector SIG. So I'm a code owner for a few components of OpenTelemetry Collector, especially around the contrib repository. So things like the tail sampling processor or the load balancing exporter, or routing processor and routing connector now and a few other things. And of course Grafana specific components like the Loki receiver and exporter. I'm also part of the Governance Committee or the Governance Board for OpenTelemetry. So I think that's my second biggest hat there. But I'm around in a making...I don't know...creating confusion in other SIGs as well.JURACI: I guess that's what I can describe.ADRIANA: Awesome. I don't know if you heard that sound. Yeah. So there is in Toronto right now, we call it like the Canadian National Exhibition. It's basically like, I don't know, like a two week fair amusement park thing. And this time of year they have like fighter jets, they do like an aerial show. And even though we're not that close to where it's at, I want to say it's about. Probably about 4 km away from where I am, but it's freaking loud.ADRIANA: And they practice around this time. And I think the air show is this weekend because Labor Day weekend. Yeah. Anyway, that's where that horrid sound came from. That was super freaking loud. So one of the things I wanted to ask in your OpenTelemetry role, so what does the governance committee do?JURACI: Right, that's a great question. In the best case scenario, we don't do anything, but we are effectively the maintainers from the CNCF's perspective. So we are the ones taking the decisions on behalf of the project, especially when it comes to official decisions. So if there are any resource requests, especially involving money, that needs to go through the CMCF, then it has to go through the Governance Committee, the Governance Board, and we wouldn't take a look at those requests and we decide, oh, it does make sense, or it is good for the project, or it is not very nice for the project to do that. So this is the very bureaucratic view of the government. So we sign the request and things like that. Practically, in practical terms, what we do is we take care of organizing our events for KubeCon, for instance. So we apply for specific places, for specific talks at the maintenance track, for instance, or the contribfest.JURACI: We applied for this KubeCon, but we also mediate conflicts. And this is, I think, the most important part of the GB, the governance board or the GC. The Governance Committee, as we usually call it, the GC. And that is whenever we see something that can negatively impact the OpenTelemetry community, it is our duty to act on it. So there was a case a year or so ago where at the beginning of the year, where there were some concerns about one specific thing. And as a GC member, we have to go and see those allegations and go and see what is behind it. And is there any concern for OpenTelemetry as a community because of that? And if there is, we have to act on. But it is our ultimate responsibility to take care of OpenTelemetry as a project and as a community.JURACI: I think that's mainly how I see the OpenTelemetry role. Of course, we also have a say, so not the say, but we have a say in the roadmap so we try to build or establish a roadmap for the project. But because the way that we structure the project, every SIG is independent, and every person collaborating in the project or with the project has the freedom to do whatever they want. We cannot just go to the Collector contributor and say, hey, Juraci, you have to work on this receiver here. That's not how it works. We have to plan ahead. Of course, as part of the GC, we have to think about it and think, when do we want to do AGA for the project as a whole? Do we want to graduate or not? What do we want for the future? And once we know that, once we have that kind of vision, project wide vision, we try to get the message out and tell other maintainers. So this is where we think the project should be going.JURACI: So can we try to make a concerted effort to get there as a project? Most of the time we fail, but I guess that it is useful nonetheless. We were able to get a focus on metrics for a few months, and then we got metrics out, and we also got a concerted effort around logs, and logs are now also out. So for the most part. So I think it does work, but it is a fair question because it's not very clear when we are out what the GC should be doing.ADRIANA: Well, it sounds like GC wears a lot of hats. There's a lot going on in the GC.JURACI: Well, we got to meet every week for about an hour, and that's pretty much all of the time that we have. So we can say that we have 1 hour of work per week. Most of the time, we discuss the whole hour. So we have quite a few things to talk about, but most of them are, I have to admit, they're kind of mundane. Right? So like a company, X wants to assess whether it makes sense to have a set of features for a specific platform, right? So then we go around and ask people, does that make sense? Does that not make sense for us?ADRIANA: So then how do you communicate, since the GC does, I guess, has a hand in the overall vision for OpenTelemetry, how do you communicate that information to the different SIGs to make sure that things move in that direction, even if you're not successful, but to at least communicate the idea, even if it doesn't necessarily get implemented, or maybe not implemented in the way that was originally envisioned?JURACI: Yeah, that's a good question. We have a diverse set of GC members. We have two people from the collector SIG. We have people that are not part of any other SIGs. We have two people who are part of the TC, the technical committee as well. We have people who are users of OpenTelemetry in ODIC. So what we do today is we use the other hats for the GC members so that information can spread around. So we have the Collector folks bringing information into the Collector SIG, but we also have GC members joining the Monday's maintainers call.JURACI: So we have a call every Monday for maintainers of OpenTelemetry. So we have a GC member joining that call and bringing the updates to that group. We also have a monthly GC plus TC call where we have a discussion between the two committees discussing things that are relevant for the project as a whole, but also making sure that information flows basically. And the TC is then responsible for ensuring that the technical direction of the project is set and followed by the individual.ADRIANA: Okay. Okay. Yeah, that makes a lot of sense. Cool. So now, taking a step back, my question is, how did you get involved in OpenTelemetry in the first place?JURACI: I like to say that I'm part of OpenTelemetry since it was not OpenTelemetry. Right. I mean, I was part of the OpenTracing group back then.ADRIANA: Oh, very cool.JURACI: I was actually part of perhaps the first KubeCon where OpenTelemetry had an appearance. And it was actually here in Berlin in 2017. I think it was KubeCon Europe was very small back then, I think not even like 1000 people.ADRIANA: Oh, my God.JURACI: Yeah. Comparing now, like with Amsterdam. It's a totally different vibe.ADRIANA: That was outrageously huge. And I think Chicago is going to be even bigger, I think.JURACI: So, yeah. I'm really looking forward to it. But back then in 2017, we had a, I think it was a tutorial. Me, Priyanka Sharma, who is now executive director of the CNCF, and Ted Young, your colleague, Ted Young. We were there doing an OpenTracing tutorial back then, and it was really fun. I mean, we had a 90 minutes session there, people trying to instrument their applications using Go and OpenTracing and facing all sorts of problems. And things were working, but barely, but it was fun. So I joined back then and things just evolved from there.JURACI: Right. So we got, in 2019, perhaps, we joined forces with OpenCensus, then we formed OpenTelemetry in perhaps a little bit later than 2019, but then here we are. And I started contributing with the Collector when we joined forces, because I thought the Collector was a really cool technology back then it was part of OpenCensus service, so it was already getting our attention at RedHat back then, we thought this is a cool piece of technology that can really free up people. Not free up, but to liberate people from vendors if they want to do so. Right. So people can start using the service to make translations and they can decide at the service, OpenCensus service. Back then they can decide where to send their data. For a company like RedHat, that made a lot of sense because RedHat was not, and is not, as far as I know, interested in a backend for telemetry.JURACI: But at the same time they were and still are, probably interested in getting telemetry data out of their OpenShift clusters or Kubernetes clusters. So it does make sense to have a support for a service of some sort, like OpenCensus service. When we had a disfusion of OpenTracing and OpenCensus, then I continued working on...well then I officially started working on the OpenTelemetry Collector. It was renamed, and that prevailed until today. I continued focusing on OpenTelemetry. And a few years ago Grafana got in touch with me and know we are highly interested in OpenTelemetry and we need someone who's already in the community to help us navigate this community and understand what's going on and bring what's new inside the company and help the company provide also support for the project in whatever way makes sense for the company.ADRIANA: Oh, that's so awesome. That's so awesome. It's nice to have your work recognized like that, where a company comes to you and they're like, hey, I like what you're doing, it's wonderful.JURACI: Yeah, I can say that I'm really blessed to work with what I really like. I really like the project OpenTelemetry. I like what I do on a daily basis. I like of course, the Collector. I like writing new components and so on and so forth. But I also enjoy the community side of it. And I think that's a huge part of what I do nowadays is building bridges. It's making connections between people and it is also empowering other people.JURACI: So helping people achieve what they want, both in terms of community and their professional goals as well. I think I'm very blessed to be where I am right now.ADRIANA: I love that so much because I think it's so important. People really underestimate the power of connections and community. And I completely agree with you. One of my favorite things is being able to connect people. And sometimes you'll have a conversation with Person A, and then you'll have a conversation with Person B, and maybe the stuff that you're talking about, it's like peripherally interesting. But then person A and person B have the thing in common and you know both of them now you can connect them. And I think it's so cool to be able to make those kinds of connections and make introductions and see the sparks fly. I think that's so amazing.JURACI: I love that. And I love seeing the results a few months later as well. And then you look back and you see, oh, something came out of that and that's really cool. So I love that as well.ADRIANA: Yeah, amazing. I just want to turn back to the OTel Collector because I'm very intrigued. I had no idea that the Collector was actually like a component of OpenTracing that got ported over to, sorry, OpenCensus that got ported over to OpenTelemetry. That's so cool. When I first got wind of the OTel Collector, immediately I decided that was like my favourite thing about OpenTelemetry. I don't know why I think it's so cool what it can do. I don't know. I love this idea where especially because so many Observability back ends, they all have their different agents, and the OTel Collector is basically the vendor agnostic agent that will do all the things and will ingest the data from different sources.ADRIANA: And then my absolute favourite is being able to send it to different simultaneous sources and to be able to see, like if you're evaluating a particular vendor, you can see how the same information ends up being rendered differently by the different vendors, and then that becomes the differentiating factor between the vendors. I think that's so cool because then the vendors don't have to compete on the data format itself. Really what's distinguishing is what do they do with the data to make it useful to you to troubleshoot. And I think that's so, so awesome.JURACI: Yeah, I think that's beautiful also for the Collector, of course. But I'm looking back the industry a decade ago, right, we see that vendors were fighting for the instrumentation. So they were saying, oh, you want the best of your services, you want the best telemetry data out of your systems, then you install my agent here. And then another vendor would just come and say, oh, you should use mine. And if you wanted both, you couldn't just use both agents. They would conflict and one would very likely have troubles with the other. They cannot run at the same time. If they did, it was not on purpose.JURACI: On any issues, any problems, you would call one vendor and they would point fingers to each other. Now the situation has changed drastically today, so today, I like to say that instrumentation is commoditized. It is not where is not where the fight is right now. Of course, there are still vendors offering their own agents and whatnot, but it's not really where the differentiation is. And it's not the collector either. Or it's not the infra that helps a data from A to B. It is really on the back end. It is really how you build a scalable back end for metrics, logs, traces, profiles, and RUM and so on and so forth.JURACI: And that's where innovation is. That's where the differentiation is right now. And I think the Collector helps people who are still on the old way of doing things, and they want to get into the new way of doing things. And it is the part that you just drop into your infra without changing anything. You just drop it there and it just works. And it will just help you achieve something today right now. So you can certainly keep using your current vendor, but you can also multiplex or send out or send the same data to another vendor. And as you said, just compare the same data visualized in different ways.JURACI: I think that's why the Collector is so it gets a lot of attention or gets our attention, right, because it is such a powerful and yet easy to implement solution that allows people to get started really quick. I mean, for instrumentation, it's nice that I can follow a tutorial on a website and learn how to instrument things and learn how to apply like Java agent or the Go auto-instrumentation and so on and so forth. But the practical results on my daily routine, they take longer to reflect than the Collector. I think that's what makes the Collector very special. And it is also the versatility of the Collector right now. I mean, just today someone was mentioning, oh, I'm playing with the tail sampling processor, and I'm applying span metrics after that. And of course, we know this is a problem, right? And the person came to that realization that it is a problem, because now I'm doing metrics only on 10% of my traces. So how do I solve that? And then with the Collector, in like five or ten minutes, I was able to get into a configuration where we have connectors and we have traces going in and connectors sending the same data to two different pipelines.JURACI: And one of them is with the span metrics, the other one with the tail sampling, and that's it. Voilà. It's like a chef of the cuisine that would just get a recipe there for a very nice and tasty dish there. So I think that's what I like about the Collector as well, its versatility.ADRIANA: Yeah, absolutely. And it's so cool to also see the different because there are so many different receivers available for the Collector now where it can ingest from so many different data formats, which is awesome. So then it's like exactly what you said. You can just drop it in. You don't have to disrupt your existing system. I think I've heard some scenarios where people were using statsd and it's like, great. You can just drop a Collector there to ingest the data from statsd until you're ready to remove that part. You can just keep it running business as usual, which is really nice.ADRIANA: And then the other thing that I find interesting and not necessarily a Collector thing, but an overall OTel thing, that when OTel started, I think there were very few vendors that used to ingest data in the OTLP format. And so there were different exporters for these different vendors. But it's been really cool to see many, many vendors now being able to support the native OTLP format, basically rendering these exporters obsolete, which is amazing, right? Because it just goes to show how many vendors are actually taking OpenTelemetry seriously.JURACI: Absolutely, yes. And it is a quite different view of the road from a few years ago. Right? I mean, a few years ago we still had vendors wondering if this OpenTelemetry thing is here to stay or not. And can I just perhaps rename my monitoring pages to Observability and be done with it? Can I ignore OpenTelemetry altogether? They realize it's not the case, so it's not enough. And they have to at least ingest or accept that OpenTelemetry exists. They don't have to be part of the community, so we don't have this requirement. They can just live on their own island. They can ingest OTLP.JURACI: That's fine. Their customers are requesting them to ingest OTLP. So that's why we are doing our customers, all of our customers, they do want to generate, know, commoditize instrumentation that we talked about before, and now they want to send data to you because you are providing a very nice solution there. And if you don't, they're not going to give you another chance. They're going to go to another vendor. And I think that's how it is today and I think it's beautiful where we are right now.ADRIANA: Yeah, I really love it. When I first started learning about OpenTelemetry, I want to say it was around 2021 and I was working at Tucows and I was running an Observability practices team there and traces had not even been in GA yet. And I'm like sitting there telling no, no, this is going to be a big thing, you just wait. And I'm so so happy to see how much it's grown since then. And now we're at a point where metrics, I believe are in GA. I think logs are stable, right? I think at this point?JURACI: The data model is. We don't have a logs API and we're not going to have one, right?ADRIANA: Right. Right.JURACI: Of course we realized, I think it was clear to everyone since the beginning, but there is an official realization that it doesn't make sense for us to come up with a new logging API. Logging is the older of all of the telemetry signals people have since they started writing code. So it doesn't make sense to try to come up with a new logging API and hope for people to use our APIs in the future. Coming from a Java road and you too, you can probably name more than five logging APIs there and logging frameworks, and we don't want that. We don't want to deal with that kind of problem at the OpenTelemetry level. So what we are doing is we have the Logs Bridge API and that is something that we can implement in every language, or almost every language, and interoperate with the instrumentation that people have today. So if they have SLF4J, for instance for Java, we can have an implementation of that for OpenTelemetry. So users still use the SLF4J API if they want and they have an OpenTelemetry implementation or the new slog library for Go.JURACI: So we can implement a handler for that. And users are just going to use slog libraries for their code. But during the initialization of the logging library we can configure that to spit out OTLP, right?ADRIANA: So basically for every, or I guess the goal at least is for every logging library out there, there's going to be like a logs bridge API that basically is that connector between taking that existing logging library and converting it to OTLP.JURACI: Of course all of the libraries out there is a little bit too much, but...ADRIANA: I guess the more popular ones I would imagine.JURACI: And I guess that goes into a nice other side discussion that should that belong to OpenTelemetry, should that kind of work belong to OpenTelemetry? Or do we expect the logging framework implementers to provide such a bridge? Right, the same for instrumentation. So do we expect database client developers to integrate directly with OpenTelemetry or do we expect the OpenTelemetry community to provide instrumentation libraries for those database drivers, database lines?ADRIANA: Yeah, that's actually a really excellent question. Has it been answered yet, or is it still under debate?JURACI: Well, it is something that we did talk about during our OpenTelemetry Leadership Summit earlier this year. And the long term vision is of course that OpenTelemetry would become so successful and so pervasive that people are going to use OpenTelemetry everywhere. And we don't need to do the instrumentation on our side. People who are domain experts and coding experts on their side, they can do the instrumentation with OpenTelemetry libraries better than we can do it. And that free us up from the burden of the burden. Burden is probably the wrong word, but the burden of creating instrumentation libraries, so.JURACI: We have to maintain them.JURACI: So we have so many instrumentation libraries out there for Java, for go, it's impossible for the limited amount of maintainers that we have to keep them all up to date across all of the versions of all of the libraries. So it's a huge amount of work, and even if things work today, we are not sure they are going to work tomorrow.ADRIANA: Yeah, that's a really excellent point. And I think it goes back to a piece of feedback that I heard in one of our OTel End User Working Group sessions from earlier this year, which is basically like, you've already got folks worrying about the API and SDK for each language, right? And that sucks a fair amount of time. But then now also having to deal with the third party libraries and not necessarily being an expert in that library, and you're having to rely on the goodwill of the community in some cases to be able to instrument those libraries, which is awesome that that sort of thing exists, but that is a massive, massive undertaking.JURACI: Yeah, it is something that we have to think about for the future. We can start thinking about that today. And there are examples of projects doing that today, like using OpenTelemetry natively. But until Open Telemetry is like the winner or the perceived winner for all of the signals, or at least for metrics as well, then it is not going to be adopted by other projects natively. So if I'm a maintainer of a project that is starting right now, and perhaps it's becoming huge success in the future, do I really want to tie my users to this library here or to that library there? And if there are no clear winners for that right now, I should probably stay out. And it's perfectly understandable. I mean, for traces it's clear. We have OpenTracing, we have OpenCensus.JURACI: They're now both OpenTelemetry. What about metrics? So I think perhaps there is a discussion to have in the future, seeing how we progress. But at least for traces, we are there. So we can start that conversation now for traces.ADRIANA: Yeah, and that's really good. Interesting point that you made on metrics, because it's a question that I've had for a while, because OpenMetrics also exists. So then what's the relationship between OpenMetrics and OpenTelemetry? Do you have any insight into that?JURACI: Yeah. So OpenMetrics, we have the Prometheus working group as part of OpenTelemetry, and I think it fits there. So we have folks from the OpenMetrics project joining the Prometheus WG for OpenTelemetry. And our idea is that we should...so OpenTelemetry should be compatible or interoperable with OpenMetrics and Prometheus. So OpenMetrics is the format, is the exposition format for Prometheus, basically. So if we want to expose data in Prometheus format, we use an OpenMetrics, or we should use OpenMetrics specification for that. I think it is the other part of the project that is the acknowledgement that we are not alone and we're not alone there.JURACI: So we have to play with the other players, we have to be interoperable with the other solutions. We have to have Zipkin, Jaeger, OpenCensus receivers for the Collector. They're not going to get away. They're not going to go away, and we don't want them to go away. It's part of a healthy ecosystem to have multiple implementations of the same solution. Same with metrics. So we want very good support for not only Prometheus, but for other metrics solutions out there as well.ADRIANA: Yeah, right. Yeah. And that's a really great point, is acknowledging that there are people still using other protocols, other tools out there, and so being able to basically welcome them into open telemetry and playing nice in the sandbox is definitely an important message to give. Now, we are just about to wrap up, but before we do that, I did want to ask if there's anything that you're working on that you would like to promote and share with folks. Absolutely. I would love to share that here.JURACI: Yes, absolutely. So one thing that I'm particularly passionate about is our participations in the Outreachy program. So, Outreachy is an internship that allows people coming from an underrepresented background in our industry in it. It allows them to get in a paid internship to work on open source projects. And since 2017, back with Jaeger and OpenTracing. I try to be part of this project. And this week we got a confirmation from the CNCF that we can have one intern working. So people who are in the industry, and if you know anyone who's having trouble getting into our industry because they are part of an underrepresented group of people, spread the word and help me find those people.JURACI: And we are being part of this program once again. So the internships should start in December and finish in March, and we are going to have projects related to OpenTelemetry there.ADRIANA: Amazing. And I also want to add that you have a weekly show in Portuguese called "Dose de Telemetria" which, if you're a Portuguese speaker, definitely check it out. I'll include it in the show notes. And also to congratulate Juraci on getting a speaking spot at KubeCon, North America in...oh my God, it escapes me. Contrib fest.JURACI: Yes, that's right.ADRIANA: Yes. Congrats. And also that Jurassi is a fellow CNCF ambassador, so wanted to throw that out. Great. Well, thank you so much, Jurassi, for geeking out with me today. Y'all. Don't forget to subscribe. Be sure to check out the show notes for additional resources and to connect with us and our guests on social media.ADRIANA: Until next time, peace out and geek out. Geeking out is hosted and produced by me, Adriana Vilella. I also compose and perform the theme music on my trusty clarinet. Geeking out is also produced by my daughter, Hannah Maxwell, who incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento me slash geeking out.

About our guest:Abby (she/her) is a Principal Engineer at Syntasso delivering Kratix, an open-source cloud-native framework for building internal platforms on Kubernetes. Her keen interest in supporting internal development comes from over a decade of experience in consulting and product delivery roles across platform, site reliability, and quality engineering.Abby is an international keynote speaker and is co-host of the #CoffeeOps London meetup. Outside of work, Abby spoils her pup Zino and enjoys playing team sports.Find our guest on:LinkedInX (Twitter)HachydermFind us on:All of our social channels are on bento.me/geekingoutAdriana’s X (Twitter)Adriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:KratixSyntassoVClusterHerokuOpenTelemetry CollectorOpenTelemetry OperatorHUSTEFCivo NavigateInstruqtAdditional Links:Abby at DevOps Days London 2023Abby's Keynote Workshop at HUSTEFAdriana’s KubeCon talk on Platform Engineering with co-speaker Ana Margarita Medina (sched.com)Adriana’s Observability Day talk on the Observability of CI/CD Pipelines with co-speaker Reese LeeTranscript:ADRIANA: Hey, y'all, welcome to Geeking Out. The podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And geeking out with me today is Abby Bangser. Welcome, Abby.ABBY: Hello. Thank you for having me. Super excited to talk about all of those subjects, actually.ADRIANA: Yay. I'm so excited to have you on. We had you for on call me maybe, and that was like a real treat. So I'm really happy that you're able to come on to geeking out. So, Abby, where are you calling from today?ABBY: I am calling in from London. Despite this accent. London, UK.ADRIANA: It's so cool to see where people are calling in from because there's always an assumption that it's like a North American focused podcast or probably American focused. And here we are. Super international.ABBY: Yeah. I've learned from living abroad that if people are speaking with the English dictionary, I'm just super content that I can understand what they're saying. And I actually find myself completely missing accents all the time where someone will say, oh, the Scottish person. I'll be like, oh, shoot. I actually don't remember what accent they had or South African or wherever. And it's just. I'm just really happy when I can speak the right language with people.ADRIANA: I can definitely relate. So are you ready for our lightning round questions?ABBY: I think I have to be. Let's go.ADRIANA: I promise they will be painless and fun. Okay, first question. Are you a lefty or a righty?ABBY: Righty.ADRIANA: Okay. IPhone or Android?ABBY: Absolutely, Android. All day.ADRIANA: All right. Mac, Linux, or Windows?ABBY: I have all of them. I've been on Mac most recently with software development stuff.ADRIANA: But which one do you like better?ABBY: It depends on how fiddly I am being. So I did the Linux desktop thing for a few years, and it is fun to get into all the configurations and setups, but sometimes you just want to print something. So in that world, I'm a big fan of the Mac.ADRIANA: Yeah, I feel you. I went through a Linux desktop phase and I was like, this is awesome. And then I'm like, oh, shit, I can't do anything. And this was back, I want to say, in 2007, where it was the.ABBY: Year of the Linux desktop. Just like every year is the year of the Linux desktop.ADRIANA: For me, the deal breaker was like, when I had a BlackBerry at the time and I couldn't use the BlackBerry app for syncing, I'm like, fuck. So then I ended up having. I had a Windows VM for a while. I ended up doing a Windows dual boot and then I moved to a Mac and not looked back since.ABBY: Yes, I understand that.ADRIANA: Okay, next question. Favorite programming language.ABBY: Oh, favorite programming language. So I'm working in Golang right now. I've always been someone who looks at code as a problem solving thing. So I haven't really gotten too esoteric about the most perfect language I've written in. So yeah, I just like to work. I guess this comes from my roots as a consultant of that I've written C sharp, Java, JavaScript, Python, Ruby, and then lots of infrastructure as code languages. And so I just sort of want to solve problems in the languages that people are working in.ADRIANA: I love it. Yeah, very nice. Cool. Okay, next one. Dev or Ops?ABBY: Got to be Ops. I guess if you make me choose. I love the impact you can have in Ops. Like just the scale of impact within an organization that you have when you have a really nice operations to the process. So I've been on platform engineering teams for, I don't know, the last eight or nine years, so it's hard not to pick.ADRIANA: I guess. Last week I recorded with Tim Banks and I asked him the same question. He's like, well, Ops runs the world.ABBY: I suspect we'll get into what is the true difference between Devon Ops at some point later today. But if you make me have to pick, I'll pick my roots in Ops.ADRIANA: All right, fair enough. Next one. JSON or YAML?ABBY: I really like comments so I think I have to pick YAML, but oh boy, those spaces. That's a tough one. I'll go with YAML for the comments.ADRIANA: Yes, I do appreciate the comments. Until someone pointed out that you can't do comments in JSON, which I knew but consciously was like, I think I've gotten angry at it before for not being able to do comments, but I'm like, oh my God. Yes. Two more questions. Do you prefer to consume content through video or text?ABBY: Ooh, I'll go with video. But I will say that what I do is I keep a folder of different content on my phone of what I want to bookmarks or whatever to watch or to listen to or to read. And I just use them at different times. So I separate them not by content topic but instead by way of consuming. So if I'm going for a bicycle ride, it might be auditory something, and if I'm sitting on the train it might be reading. And so I do like a bit of both.ADRIANA: But yeah, fair enough. Yeah, that's a really good point. Yeah. I was telling someone today, I love the audio stuff for when I'm doing busy work chores around the house or what you might call it, like going for a walk or for a run. Especially when I'm running. I hate and love running and it makes the time pass faster.ABBY: I can answer that one much faster than all these. Hate, hate running. That'll be the quickest of all these lightning round questions.ADRIANA: Final question is, what is your superpower?ABBY: What is my superpower? I'm going to go with being able to energize other people if I'm going for a real one. I'm coming off the back of a very exciting tag rugby win last night for a team, and I unfortunately have a bad back and couldn't play. But the team just rallies in a way that I feel like I can help bring energy to the team. And so I did my part despite not being on the pitch. Yeah, I think that's probably it.ADRIANA: That is a good talent and very translatable into the work world as well, right?ABBY: Yeah, I think it works out okay. I mean, I'm sure there's times when it gets annoying. I'm very sure that's what all superpowers, right? They have their good and their evil. But I like to think it's more good than.ADRIANA: When I remember when someone mentioned, oh, check out Abby's content. And I started watching your videos and reading your blog posts, and I remember you just came off with such infectious enthusiasm and just like, oh, this is somebody I'd like to chill with.ABBY: And then we got to. So it was great.ADRIANA: We get to see each other at KubeCon in November again, which I'm super excited for.ABBY: So excited for a trip back to Chicago.ADRIANA: Yeah, yeah. I really like Chicago, and I never spend long enough in Chicago to actually see it properly. It's the curse.ABBY: Yep. The curse of traveling for conferences is that you're always like, oh, I'm going to this great city and I will see the convention center.ADRIANA: If I'm lucky, I'll get to see the bean again.ABBY: Yeah, exactly.ADRIANA: Thanks for answering the lightning questions. So onto our regular business. So one of the things that I wanted to talk to you about is platform engineering. It is the hot topic these days. So I think, for starters, why don't you explain to folks in your view, what platform engineering is and how it differentiates from DevOps and SRE, for example.ABBY: Yeah. So platform engineering is, in my opinion, and from kind of taking together a few different quotes from other things, it's a way of building internal services to support kind of the business use cases around an organization. I think that right now there's a lot of focus on how that impacts things like developer experience for software engineers. But I'm very specifically, I feel like the definition is a bit broader than that. It's about supporting the delivery of business value at scale with consistency and security and compliance in mind. And platform engineering is the act of putting that together into services that are consumable by the organization for those end goals. I think I have a talk at DevOps Days London, where I do just boil this down to it's internal services teams done better. So it's internal, it done with more kind of product mindset in a lot of ways.ABBY: And I think where you start to differentiate between some of the other kind of buzwords and titles and philosophies that are out there, depending on what they are, is the way I've been looking at platform engineering recently is that it's an implementation of the DevOps intention. So with DevOps intention, it's to reduce the friction between silos. You looked at silos of software development and software operations and you wanted to improve the friction between that and the quickest way to do that. The best way to our understanding, was to remove the silos, ask software developers to understand their operations, ask them to figure out how to do on call and telemetry and all of these things, and simultaneously ask operations engineers to learn about software development and to bring automation to their processes and testing and all those kind of things. But that's hard. That is really hard because you're prioritizing the removing of friction over the depth of knowledge for certain people. And that sort of specialization value that comes from someone who deeply understands an area of the business or of the technology. And platform engineering, in my opinion, is also really hard just to say, but the idea is to productize those specialties and make it so that the silos can still exist.ABBY: But you're focusing on the friction between them that's being reduced. So you still have your specialists, but they're providing a service to end users that they think about like a product, and they provide in a way that is easy to access and easy to use.ADRIANA: Yeah, that sounds awesome. That makes a lot of sense. Now, follow up question. In your opinion, do you think that we've achieved what DevOps promised?ABBY: So DevOps basically promised the idea of reducing friction on the software delivery kind of process or lifecycle. I think we are achieving that. I think we have less friction than when DevOps was born 1015 years ago, twelve years ago, whatever. Do I think that we are going to get more return for the experiences and processes we've put into place over the last ten years? This is where I think we're seeing a shift in focus. So the intentions of DevOps, we are doing better, we are moving our way, but if we keep kind of trying to drill into this idea of everybody builds and runs their full stack, we're going to lose the fact that, no, we don't. First of all, we don't plug in our own hardware. Most people, most people aren't both wiring up a server and writing the CSS for their front end. Their full stack is whatever their team or their company defines.ABBY: And I think if we continue to think about the friction between the layers of value that we're providing and we keep trying to reduce that friction, we'll keep gaining benefits on this journey of DevOps. I don't know if there's an end state for DevOps.ADRIANA: And that makes sense because one of the things around DevOps is like this idea of continuous improvement and there's never going to be like perfection, right? We always chase it, but we aspire to it. Software is never done.ABBY: No. And that's what we're doing with platform engineering, bringing development to operations. We're making more software and we know software is never done. It always needs upgrades, it always needs improvements, it always needs coming back to, to make sure we're not customizing things that are now commodities. Right? So there's lots of things that you may have written a few years ago that now there's public packages to lean on and so you don't want to be maintaining that yourself. And if you don't do that, you very quickly get snowed under with the amount of maintenance you have on things that are not actually differentiating for you, your team, your business, your organization and all that. We're just multiplying software everywhere. And so we absolutely need the DevOps principles of reducing friction between teams and building and running the software that you are owning so that you have that feedback loop from production into your development lifecycle.ABBY: I think we need that as much as ever.ADRIANA: Yeah, I totally agree. And you bring on a good point in terms of like you may have developed this on your own in house at some point in time, but if there's something better that came along that solves your problem and then some, it ends up being way more awesome. But I feel like, and I fall into this trap before as well as a developer getting into this idea of like, oh my God, I get to build something cool and new and this is awesome. And then you build it and then you're like, shit, I have to maintain this. And then the shiny and new thing kind of just wears off and you're like, never mind.ABBY: Well, this is the problem with the average tenure of employees being like two years, right? By the time the average cycle is you come in excited about your new job, you spend then maybe a few months getting up to speed with how things work and what's going on, and then you've got maybe about six months to a year of whinging about something that you hate about the way that things are done wherever you are and advocating that something change. And then you implement that change. And then you move companies or you move teams and it's like, well, there's somebody doing the things that you're complaining about as you go to these new companies. And I would hearken to guess that the people who come in after you have plenty of complaints. And it's just that learning about what creates that maintainable and kind of experience that you want to have as maintaining software is hard to learn if you don't spend years and years maintaining the same software. And yet that's just not how our industry right now happens for many people.ADRIANA: Yeah, it's so true. We get bored easily or we're chasing the next shiny thing. Ooh, I get solved this problem. That sounds way cooler.ABBY: Yeah, exactly.ADRIANA: I guess the good news though is that there are companies out there that are working to alleviate some of the problems that developers and platform engineers and Ops folks are experiencing. And we've seen that with the umpteen different tools built to support Kubernetes, various platform engineering tools and whatnot. So it's been kind of interesting to see how much the market has evolved in the last 20 years.ABBY: Yeah, it is amazing. And I think you say 20 years and that's absolutely true. But there's been even more explosion in the last maybe handful of years, five years or so. As you say, platform engineering is everywhere. And the reason it is is because that feeling or that interest in developer experience has grown tremendously over the last kind of four to five years. And platform engineering is one way to possibly address this application software development experience. And so yeah, I think you're seeing all these tools that are building off of the PaaS experience, the platform as a service experience that was, I think, first really popularized by Heroku in kind of the 2012 ish era, but now that's just commonplace that there's lots of tools out there that act in that kind of way.ADRIANA: Yeah, so true. And on the platform engineering front, I wanted to ask, what do you think are some of the biggest problems that platform engineering is trying to solve right now?ABBY: So if you listen to marketing, it's all developer experience, all day, every day. It is. How can the application software developers find what their applications are doing? They need a portal and they need to make their lives as perfect and glorious as possible. But I don't actually think that is the case. I think that is a piece of things that we're trying to solve with platform engineering today. But when you talk to the companies that, I don't know, make money in things, the big organizations, they're actually not always focused on developer experience as a top reason for platform engineering. Often they have issues like compliance and security and regulation, standardization and support as a key reason. Often they have cost optimization as a key reason.ABBY: There's all sorts of reasons why providing a centralized offering as a service can benefit the business and is only auxiliary, supporting the app dev experience. But that isn't the reason for the work being done. And yeah, I don't think that gets enough press, right, because it's not as much fun as making a developer portal and things.ADRIANA: That's so true. Yeah. I mean, every time we talk platform engineering, I feel like developer experience is almost synonymous with it. And then all this other stuff takes a backseat. But compliance standardization, so important, especially, obviously in any size, but especially the big corporations that have to worry about regulators and compliance and all that scary stuff, right? I mean this is serious business. So being able to cater to that, cater to security concerns, having a standardized way to ensure that things are locked down to your company specifications, so important. And as you said, not talked about enough.ABBY: You've tripped right over one of my biggest pet peeves as well, where you mentioned that developer experience is like synonymous now with platform engineering in some ways. And also developer experience at this point is synonymous with a software application developer's experience. So someone who is writing a web app or a backend app, API app, something that is like Java or whatever, running in a container or on WASM or wherever, but the software side, they aren't the only developers. As I said in the Lightning round, I pick ops like I'm a developer, even though I write HCL code, even though I write bash, even though I write whatever chef recipes and whatever else on the back end. So I think the fact that developer experience has then I would say narrowed to the point where it's only application developers is showing, I think, in that explosion in kind of tooling and activity in the industry, because you look at all these kind of products that are coming out and they're all focused on how to get a piece of software out in front of an end user as fast as possible. But when you start to dig into how do I operate this thing, how do I build in my PCI compliance and my workflows and my marketing workflows and my customer success workflows and all these other things I need around the business, it all falls apart. And I think that too many of these forget that there's a whole other side of software delivery that isn't just writing software code. And that experience needs just as much kind of improvement, if not more.ABBY: Because again, the leverage of that across all your software teams is huge. So yeah, it's one of my little pet peeves. You chipped right over it.ADRIANA: Moral of the story, get some TLC. Outside of just developer experience, just app.ABBY: Dev experience still can be developers even, but yeah, even that's probably too narrow. I agree.ADRIANA: Yeah, very good point. Yeah, and you make a very excellent point also. You're a developer regardless of whether or not you're developing the application software or you're developing code to operate your system, it might be slightly different as to some of the things that you're concerned with, but at the end of the day, code is code. Technical debt still exists in both realms.ABBY: And the tools you have impacting what you can create exists in both realms. I think I've been on the software side for many years as well. And on the software side, I've been in a situation where an internal team provided an API that my team needed to use to create a visual front end for. And the shape of the data that came back from that API absolutely influenced and even kind of forced our hands sometimes in just what we could create on that user interface experience. Because not being able to receive the data in certain ways or access it on certain pages or whatever, I mean, the same thing goes from the platform engineering standpoint. The tools we have, the baseline kind of bash and Goling and Ruby and whatever that we're using are forcing our hands to create what I think is actually not create developer experiences. We're exposing application developers to helm charts and terraform modules and chef recipes. And it's like they shouldn't have to know what languages we write, they shouldn't have to know what tools we rely on.ABBY: But because they're all so primitive and they're all so low level. We're working so hard to piece these bits together for our business process that that's like as far as we can get on the platform engineering side or it feels that way. And so I think getting more, as you say, TLC to those platform engineers is only going to make better experiences for the application developers because they'll start to have a bit more bandwidth to be like, hold on, how are people actually using the stuff that we create? Is this the experience we want to give them? Is it about cloning a git repository or forking a git repository or something? Or should it actually be a different experience altogether? Something as a service behind an API or something else?ADRIANA: So here's a question. How do you ensure that you bridge that gap to raise the awareness so that it's not just this developer experience, application developer experience centric view of platform engineering? How do we shift the conversation?ABBY: I mean, we just need platform engineers to speak more about what they need and sort of get involved in the community. So one of the things I'm doing is I'm a part of the CNCF or Cloud Native Computing Foundation working group model. So the way it works is that there are groups with different focuses and they sort of narrow as you get down the model. And I'm in one of the kind of most narrow things, which is called a working group for platforms, and this is under the umbrella of app Delivery technical advisory groups, or tAg. And the idea is you want to be able to deliver applications. And one of the ways in which an organization can support app delivery is through the creation of a high value platform internally. And this working group is having all these conversations about identifying ways to speak to business leaders, CIOs, CTOs, CEOs about the value of investing in platform engineering experience and tooling to be able to support this kind of outcomes for them. So, yeah, get involved in the community, start kind of helping shape the conversation a bit more.ABBY: Right. There's, I think, less platform engineering voice than there is application developer voice right now.ADRIANA: Yeah, absolutely. And one of the cool things about the working group that you're part of too is it's representation from different platform engineering vendors or I guess practitioners. So you get different perspectives, different voices, sometimes competitors all working together, right?ABBY: Yeah, it's absolutely. I think there's a heavy number of vendors and a heavy number of kind of consulting oriented people, but there's also a heavy number of kind of end user related people as well. So people who work at these large, successful enterprise organizations and are driving these initiatives internal at those organizations come and speak with the working group about their experiences, about where they're feeling like they need more support from kind of the industry or from tooling and from vendors, as well as how they're piecing together these tools to create the experiences that they are. And we always want to welcome more people with more stories. I mean, we're working right now on a follow on from a white paper we released in the spring, which was around the platforms as a definition, as a white paper. We're now working on making that a bit more tangible, with a maturity model to talk about how you can kind of evaluate and grow your experience with platform engineering in relation to that white paper. And we've been through one round of edits already, and I think we have something like 30 or 40 collaborators already, and we're going into our final edits now, and we're only seeing new people pop up every day. So I really believe this is going to be a global vendor agnostic, business domain agnostic piece of work that will hopefully help people have these hard conversations at their organizations and get the kind of support they need to be successful with platforms.ADRIANA: Oh, that's really nice. That's really nice seeing the community come together like that. And what are some of the, I don't want to say grievances, but I'm going to say grievances that some of the practitioners come with when they share their experiences.ABBY: Yeah, I think that it's often just not knowing what tools to grab for and how to interrupt them all. So each tool, it's in its own right, is interesting and exciting to them. But right now, there's not enough of an ecosystem around platform engineering. Everyone's sort of fighting to be like the one tool that rules them all and says, if you use us, you don't need to worry about anything else. And the reality is that most of these organizations really, at any scale, from quite early on, will have a fairly diverse need, right? They'll have front end apps and back end apps, which will often lead to different languages, though not always. They will have applications that are more on the cutting edge and are their kind of exploration applications versus their more money making applications. They'll have ones that are compliance related and ones that aren't. And so there's just so much nuance and diversity, and yet people are trying to still sell this monolithic idea of the one platform that you can kind of buy and use, and it's just not being realistic based on the feedback we're getting from people.ADRIANA: It would definitely be lovely to have.ABBY: That if you can do it.ADRIANA: Yeah, that is a lot of stuff going on, a lot of different needs to cater to, which makes it very challenging.ABBY: Look, if you can start as a small company with using a purchased solution and keep everyone on the rails on that solution, you will be very happy. Right? That is absolutely the cheapest way of doing any of this stuff. But the way that I try and describe it is it's basically like saying that you can use another company's processes for your company, and as soon as you have chosen to create any processes out of band from that kind of platform delivery mechanism, all of a sudden you just start that fissure and you just start that divide of what can be done on platform and what has to be done off platform. But again, if you start early with a single kind of monolithic platform as a service and you can keep everybody bought in that that's the right way to go, then yes, that is the cheapest. It's always cheaper to lean on a vendor and lean on a commodity product that has its own set of engineers who are supporting and improving the feature set than to try and build it yourself and maintain it yourself. But yeah, again, we're just seeing that that isn't proving to be long term viable for many organizations.ADRIANA: Yeah, that makes a lot of sense. Now, what about taking the angle of even though each organization is different and has its own specific needs, we often find ourselves when we're moving from job to job that we keep solving the same problems over and over and over. And that gets kind of annoying because it's like, well, but I just spent like two years figuring this out of my old and now I have to do it all over again. And that sucks. So how can platform engineering help us with that?ABBY: Yeah, I think there's kind of two sides to that, or two things I'm seeing happen right now that might help with that. It's hard without a glass ball, right? But one is that you look at the kind of ecosystem that grew around docker containers and grew around helm charts and grew around operators and things like that, and you can see how they're solving some of those problems in a community way, in a publicly available way. And I think there will need to be some sort of an ecosystem that continues to build those business solutions that can be shared across. But where it becomes difficult is once a business has solved it, how much do they invest in being able to make that public? It's the same idea as when you really want to make that kind of utility. You created open source and your company is like, but now you have to scrub the commit history, you have to do all these, create a contributor guide, you have to do all these things. That's true for all these solutions as well. So it's not always cut and dry. But I think that we are building ourselves bigger and bigger abstractions and maybe we will get ourselves to a point where we can provide those higher level kind of business cases out.ABBY: The other side that I'm looking at is that we've all been here with all of the things that end up becoming commodities, like the number of times we tried to recreate the wheel on creating a user facing developer portal before Backstage came around and gave us a framework for doing that. The number of times we recreated the wheel with creating web applications that could receive HTTP requests and speak to databases before we got frameworks like rails or spring Boot or pick your language, pick your framework. We see that in order to coalesce around a common problem, there have to be a lot of attempts at solving it to really understand that it is the same. And I think you're 100% right that we're getting there with platform engineering, we're getting there with the fact that hey, we want to be able to give databases as a service, hey, we want to be able to give development environments on demand per pull request. We want to be able to update the secrets that our web app depends on as a service. There are certain capabilities that I've provided time and time again across organizations and I think we're getting there where we're going to need to do kind of more of a framework approach to things, right?ADRIANA: Yeah. And I think that's really exciting prospect, especially when you think about things like renewing certificates, updating secrets. I've heard horror stories of certificates. There's one story I heard, I don't know how true it is, but at one of the companies I used to work at, a certificate expired. The guy who was in charge of the certificates was off on a cruise. Unreachable, not good.ABBY: And it's probably one of those ones that you have to email someone and then they send you back a login to a secure portal to go download the certificate to then SCP it to the server. I've been there, thankfully not in an emergency situation, but I have been there.ADRIANA: To be able to have tooling around that where you can just alleviate that problem significantly, because nobody wants to have to keep remembering how to do this. It's annoying, it's stressful. And sadly we tend to wait until.ABBY: The last minute for it's because we have so many other fires to put out. But I mean, again, your Cert example is exactly why something like Cert Manager has become ubiquitous, right? And for any organization that hasn't moved their custom bespoke process over to something like a Cert Manager is they're being tied down and they're being weighted down by this need to maintain. Something that was very necessary before and was good that they created it. But now the best action is to move forward and start to lean on the commodities. I think I'm having a whole conversation right now with a few people about this concept of thinnest viable platform that was discussed in the team Topologies book, and the fact that right now most people are defining that really as just another word for minimum viable, as in how to get started quickly. You get started quickly with great docs, I love that. Yes, get started quickly by documenting and coalescing what you have on your platform. But call that your minimum viable.ABBY: Your thinnest viable is always having an eye towards what can you offload to a third party that you no longer need to maintain. So it's not just about starting small and growing big, but it's about starting small and staying small and figuring out how you can continue to stay at only the highest level of value that you can provide and lean on as many third parties. Open source vendor I don't care to do the things that your company is not differentiating on. Lean on the industries for that.ADRIANA: I like that a lot. It's time and money well spent. In fact, it saves you time. Maybe not money, but if you end up having to fork money over to have somebody else deal with it, because as you said, it's not part of your core competency, then so be it. Because chances are they probably do it better than you anyway.ABBY: And that's exactly because just to say money is time. I was just reading a fantastic thread about technical documentation writers, or technical writers versus software developers, and how it was somebody's personal experience where they witnessed technical writers being let go for cost savings because the software devs could write their own dOcumentation. And the thread was all about this like cool. So what you're saying is that these people who make not even half as much like the tech, the documentation people, do not make nearly half as much as the software developers. You're saying it's cheaper to let them go and take the amount of time they spend on documentation and put it onto the plate of people that cost twice as much and aren't as experienced and therefore are likely going to be slower at the job than this. So you're paying somewhere. It can be off of the books, out of your bank account if you've got the ability to do that. And if you don't, sometimes you have to pay in, we call it like sweat equity or something like you have to pay in engineering time, but you're paying somewhere.ADRIANA: Yeah, absolutely. It's funny how people don't consider, it's basically a cause and effect thing, right? It's like, oh, of course it's going to be cheaper because we've got fewer people on payroll and something that someone else can already do well, that's not normally part of their job. Now you're giving them more work, you might have to pay them overtime, or you're burning them out, both of them kind of.ABBY: Or you're getting less from them on what value you expected from them than you were hoping. Right. And that might be what, you might have made that decision quite intelligently and carefully. Or it could just be that you're learning that the hard way. The side effects of that.ADRIANA: Yeah, absolutely agree. So, switching gears a little, well, I guess related but unrelated, tell us a little bit about Kratix.ABBY: Yes, I probably should disclaimer it as talking about frameworks. So I am working for a startup called Syntasso and we're building a framework called Kratix. And the idea is that pet peeve of mine about how platform engineers just get completely ignored in the developer experience kind of conversation is what we're trying to solve at Syntasso, they absolutely won me over as one of the first tires after the founders when they talked about being in the platform engineering domain and thinking about how platform engineers can thrive, not how the end users of the platforms get benefit. And I just don't think there's enough of that conversation. And this is the business domain I'm most interested in right now. So when we think about how do platform engineers thrive, I talked a bit about what I'm seeing in trends and that is what we have decided, sort of our first product to start to create. So it's an open source framework for creating business capabilities. So you think about a platform as a service, as a kind of a monolithic thing you might be able to configure, but everything has to sort of be delivered the way that platform that you purchased or that you're using works.ABBY: Why people love that experience is because they want to have something as a service. Super easy consumable don't have to think about the maintenance behind the scenes. Someone else does that. Why they have moved away from products like Heroku and others is because they don't fit their business processes. With Kratix, the idea is that you can build anything as a service for your company relying on the tools you already rely on today, but leaning on the framework to be able to provide that sort of user interface, user interaction, that automation of the workflows, that scheduling to your different destinations, your different infrastructure, and kind of remove some of that heavy lifting from your experience as a platform engineer.ADRIANA: That's awesome. Yeah. And disclaimer, I have played with Kratix before. I wrote was a promise for installing the OTel Operator so that it sends traces to Jaeger.ABBY: Yeah, absolutely. Our first community promise provided. So first of many. So yeah, it was fantastic.ADRIANA: Yeah, that was so much fun. And it was fun because I got to provide feedback on the product on something that was so new because I think you were mentioning that whenever there were promises for other tools, you guys wrote them yourselves. And I basically sat down with the docs and bugged you all a bit here and there to write this promise. So that was a fun experience. And it was also like my first experience with the hotel operator. So I'm like, what should I do? Well, let's combine platform engineering and hotel, because that'll be fun.ABBY: Yeah, it was a great example though, right? Like that OTel Collector is itself a community provided tool that can significantly help internal organizations collect telemetry and therefore have Observability. And that's great. But where do you need this thing? You probably need it in at least more than one cluster, maybe more than one namespace within that cluster. How do you manage the deployment of that? How do you manage the upgrade process of that? Often you don't just have an OTel Collector, you have a fleet of hotel collectors, and you want them to be managed as a fleet. But our current tooling doesn't do that. Our current tooling is probably customized folders or Helm charts being installed in multiple places or something to that effect. And we really want that fleet management of OTel Collectors on demand and as a service, that's what you provided through that promise with Kratix.ADRIANA: Yeah. So that was lots of fun, and I can't wait to play around some more with Kratix. I think the next experiment which you guys helped me out with was trying to install VCluster using Kratix, which I'm like, this feels like such a fun little project to come up with. And then that was neat because it kind of exposed some stuff that I was trying to do stuff with the product that wasn't available at the time. But then it's like, okay, well now I've provided you a use case for things that people might want to try out, right? Which I thought there was lots of fun, being able to collaborate with you guys on that, just being able to be part of providing meaningful feedback to a young product that's like, it's come along quite nicely since I started playing around with it.ABBY: Yeah, thanks for that. You definitely weren't the last person to have that use case of wanting on demand clusters. And I think we talked about that platform engineering right now is pretty tunnel focused on application developer experience, but actually there's so much more to it, like cost minimization and eco impact minimization and things like that. And VCluster is a great example of where you can use that product to try and reduce the number of Kubernetes clusters that go up with all the kind of redundancy on the master nodes and all those things. And so what we did with that is be able to make on demand environments in VCluster for people. So instead of giving people a whole new cluster, you give them a VCluster. It's sandboxed, it's safe, it's less impactful on the environment, less expensive on your pocketbook. Yeah.ABBY: As I say, you weren't the last. And we're working with some design partners right now. We have a few open spots. If anyone else is in this space and is interested in working with us on the product development stuff, it's all open source, Apache 2, licensed. We're not building in private here. We're trying to build for you and for the industry. So yeah, come have a chat with us anytime.ADRIANA: That's awesome. That's awesome. Well, before we wrap up, do you mind sharing with some folks some of the things that are coming up?ABBY: Yeah, so I'm not sure exactly when we'll be airing. So some of these things might be in the past, but over the autumn I'm speaking at a few events around London, which is really fun. I love being able to not have to travel and so I'll be at a AWS day and SiVo navigate and DevOps days London. I will also will be traveling to Hungary for a Housteff, which is a Hungarian testing forum, which is a nice return to my roots. I was originally a software tester and so it's nice to be able to go and connect back with that community and then I will be in November in Chicago, as we said, for Kubecon, which is super exciting. So definitely looking forward to connecting with people there.ADRIANA: Very nice. And the thing you're doing in Hungary, that's the tutorial with TraceTest.ABBY: It is. Speaking of an amazing kind of feedback loop with a great new tool. It's not that new actually anymore, but they still are just so responsive to any questions and ideas that we have that I have. And so yeah, I'm doing a keynote about observability and have created a new tutorial around using observability with trace test for the testing audience because I think that will really kind of connect people who aren't typically in the observability space but are in the testing space with connect those two dots together. So yeah, really looking forward to that.ADRIANA: That's awesome. Yeah, I can't wait to catch the recording for that. I hope they'll be recording.ABBY: I don't know if they'll be recording the tutorial, but I am doing it on instruct so I can definitely give access to it. It's all kind of open source tech that we're going to be using. So the team at Instruqt have been so good about providing access for the community driven activities like this, and they just have such a great platform for creating content and so really makes it a pleasure to deliver.ADRIANA: That's amazing. Good to know. Thanks so much. Well, we're just wrapping up, but before we do, do you have any parting words of wisdom or hot takes to share with our audience?ABBY: Parting words of wisdom, I guess when it comes to platform engineering, just remember it's an engineering, it's software. What do we need to make great software? We need to think about how to build and run that software, and we need to think about the end users because it's a product. And so whether our end users are our colleagues or our friends and family or someone else, they're still customers. And yeah, it's a product. So I think that's really the shift in focus with platform engineering in my opinion.ADRIANA: Awesome. I love it so much. Well, thank you so much, Abby, for geeking out with me today. Y'all don't forget to subscribe. Be sure to check out the show notes for additional resources and to connect with us and our guests on social media. Until next time...ABBY: Peace out and geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music, Trusty clarinet. Geeking out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout

About our guest:Ori Shoshan is the Co-Founder and CTO of Otterize, where he spearheads a leading Workload Identity and Access Management platform that automates and simplifies access control for cloud-native environments. By offering a declarative and zero-trust approach to access management, Otterize empowers organizations to streamline network policy management while ensuring maximum security.Drawing from a remarkable 15-year career as a seasoned platform engineer, Ori's expertise is underpinned by his leadership roles in both technology and personnel at esteemed institutions. Notably, he made significant contributions at the IDF cybersecurity unit 8200 and served pivotal roles in cybersecurity and developer tooling startups, such as Guardicore (now part of Akamai) and Rookout (now a part of Dynatrace).Find our guest on:LinkedInX (Twitter)Find us on:All of our social channels are on bento.me/geekingoutAdriana’s X (Twitter)Adriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:OtterizeMicrosoft IntelliMouseBlue Screen of Death (BSOD)MacOS Kernel Panic (Mac equivalent of BSOD)Mac Spinning Beachball (or Pinwheel)Yeti (mythological creature)Python async awaitHCLINI fileMSDNOtterize Client IntentsOtterize Network MapperNetwork Mapper OpenTelemetry PROtterize CLIGraphvizOtterize CloudClay Smith (OTel PR contributor for Otterize Network Mapper)KomodorOpenTelemetry DemoKubeshopTracetestAdditional Links:Find the Otterize team at KubeCon NA 2023 in Chicago, in booth P18.Adriana’s KubeCon talk on Platform Engineering with co-speaker Ana Margarita Medina (sched.com)Adriana’s Observability Day talk on the Observability of CI/CD Pipelines with co-speaker Reese Lee (sched.com)Transcript:ADRIANA: Hey, y'all! Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada.And geeking out with me today is Ori Shoshan of Otterize. Welcome, Ori.ORI: Hi, thank you for having me.ADRIANA: Super excited to have you here today. Where are you calling in from today?ORI: So I'm actually in Berlin, but normally I am in the periphery of Tel Aviv, Israel.ADRIANA: Cool. Well, let's let's get started with our lightning round questions. Are you ready? Okay, first of all, are you left-handed or right-handed?ORI: It's a complicated answer. I know you said the lightning round was going to be easy.ADRIANA: It's all good.ORI: I'm left-handed on the computer, but I write with my right hand. You can thank my school for that. They basically made me use my right hand, so I guess I'm ambidextrous.But I tend to write...like, I basically just sign stuff with my right hand at this point in life because everything else is on the phone or typed, so yay.ADRIANA: Oh, yeah, it so you were probably born a lefty, but turned into a righty.ORI: Yeah, but I like my left-handed mouse.ADRIANA: That's interesting. I'm left-handed and I never got the left handed mouse thing, I think because the first time that someone presented a mouse to me was, like, a right-handed mouse, so I never even thought of holding it with my left.ORI: Yeah, that's a big problem. I always have to like when I buy a new mouse, sometimes I feel like as the years go on, it gets harder and harder to find, like a symmetric mouse that isn't specifically for right handed people.ADRIANA: True. That's true. Actually, you just made me think of, like do you remember? I think it was like the Microsoft mouse that was like it was shaped for right-handed people, and I remember thinking, oh, my God, if you're left-handed, you're screwed.ORI: Exactly. Yeah. So I make do with the smaller mice. But it's nice. It's an excuse to buy a gamer mice sometimes. Yeah, but it must be hard to find a cool mouse that will either be ambidextrous or tailored for left-handed mousers. Yeah, but it's a once in a few years thing. I just wish it was more like headphones, where you can basically buy the same pair again and again when they go bad.ADRIANA: Yeah, that's true.ORI: As long as they don't discontinue it.ADRIANA: Yeah.ORI: With the mice every time, I'm like, I want to buy this mouse again and they don't make it anymore, so I have to find the closest one.ADRIANA: Yeah. I feel you. I feel you. Okay, next question. iPhone or Android?ORI: Android,ADRIANA: All right. Mac, Linux or Windows?ORI: Honestly, Windows, even though I use Mac daily because I've given up because everybody else at the company wanted Macs, so I didn't want to be the odd one out, but I'm a Windows guy, through and through.ADRIANA: Interesting, interesting. I grew up on Windows, and then the first time I used a Mac, I hated it so much, and now I'm a full Mac convert. But I remember when Windows came out, I was like, this is the greatest thing ever.ORI: I think the thing that gets me is like, I've been in software development a lot of years by now and so on Windows, I really know how things work, like, beneath the outer shell, and when something doesn't work in the system, I know how to debug it. But with MacOS, my understanding only goes so far. So it's like if you see the famous beach ball spinning and something stuck on Windows, I would know how to figure out what is doing that. But on Mac, I feel useless. I'm just like, okay, better force close it and try again. Pray that it fixes itself.ADRIANA: I find that the funniest thing on Mac that ever happened to me is when I got, like, the Mac's equivalent of the blue screen of death, but it was so polite, and I'm like, oh, shoot, when this happens, I know I'm in trouble. I think I only got it once, but I was like, Damn, I should have taken, like, a picture of it. I don't even know what it looks like.ORI: I think I've had a few the last couple of weeks because I opened my laptop into, oh, everything's closed, and by the way, we've crashed. So you want to send an error report? Maybe, but I didn't get to see the actual crash screen.ADRIANA: I got to find a picture to include in the show notes. I'm sure someone's taken a picture of it.ORI: It's hard to see. They don't let you see it.ADRIANA: Yeah, that's right. It's like a Yeti sighting.ORI: Yeah, exactly.ADRIANA: Okay, next question. What's your favorite programming language?ORI: It changes along the years. Right now it's Golang definitely. I like to say it's a language for stupid people, and I'm one of those stupid people because it's really prescriptive about how you should do things. Generally, most of the time, there's only one way to do something, and I find that really valuable. I think it's insane that I can look at our own code base, like the back end and our open source code base and the Kubernetes code base, and they look pretty similar. And other languages not so much. I think Rust code bases will look very different. Don't get me started with C++, where the language basically it's worse than JavaScript. The language reinvents itself every couple of years. And there was so much sorcery to know. I probably couldn't read C++ code by now, even though like six years ago it was what I was doing all day...ADRIANA: Oh, wow.ORI: ...and before that it was Python.bPython will always have a dear place in mybheart, but it got complicated in Python free whenbthey added async await, it got kind of fragmented.ADRIANA: Yeah, fair enough, fair enough. Yeah, I do agree with you. Python...I have much love for Python. I started out in Java, and Java...I don't know, it's like coding in spaghetti. It's, like, such a verbose language, and most of my career was Java, but I'm like, nah, Java, I'm over you. I do like go. It's like, generally succinct except with error-handling. It's kind of weird.ORI: Yeah, it is weird. And it's a common refrain. I feel like a macro could help there, but I actually like the concept of like no, this is how you do things. And there are clear downsides to using Panics. You could use them theoretically as exceptions, as you would use exceptions in Python.ADRIANA: Yeah.ORI: But I like that they say this is how you do things, just do it this way. Shut up.ADRIANA: I do like, prescriptive things in that manner. I think we can do with more prescriptive languages, to be honest.ORI: It's just so opinionated. And at the same time, they leave some stuff out. Like it's not so batteries included that you don't have to do anything, which is a strange combination, but I feel it works well.ADRIANA: Yeah, definitely. All right, next question. Dev or Ops?ORI: So can I say both? I feel like probably a lot of people said it.ADRIANA: Yeah, a couple of people have said DevOps, so I'm like, yeah...but it's okay to have a preference of one over the other or, like, both. It's totally cool. Like I said, no wrong answers.ORI: Yeah, I guess both, you know, nowadays it's the popular thing to DevOps. And if you talk to people who work at, like, I think at Apple, they still separate dev and SRES a lot. And I talked to people who said they basically just throw stuff over the wall, over to Ops and they get to deploy it. And I really enjoy looking at the entire stack. So DevOps is for my spirit. I like seeing things happen end-to-end and like understanding, you know, like I said about Mac, that I can only debug so far. So I like that about DevOps, that I know what the code is and where it runs and when something strange happens, I can understand the system as a whole. That's really appealing to me.ADRIANA: Yeah, I can totally relate to that. Awesome. Okay, next question. JSON or YAML?ORI: JSON yeah, the spaces. The spaces. It's also like that spaces or tabs being part of syntax.ADRIANA: Yeah, that should be a question I could ask: spaces or tabs?ORI: Another thing I hate about Python is just there are so many discussions about that, like should we use this or that in Python too? And it's like it's meaningless and so many stupid bugs just by missing one space. No, definitely JSON even though it's annoying too. But I don't think that there is an ideal choice.ADRIANA: Yeah, fair enough. Fair enough. I did have one person say HCL, which I'm kind of down for because I feel like if JSON was, like, a little more pared down with some of the love from YAMLYeah, HCL is like a bit closer to INI files, right? INI files.ADRIANA: Yeah, true.ORI: Yeah. It is simpler to parse. I feel like I got to use them in the Windows part of my career.ADRIANA: Yeah, nice. All right, two more questions left. All right, so do you prefer to consume content through video or text?ORI: Definitely textADRIANA: That seems to be the winner so far. I've had some where it's like it depends, but it's mostly text. Word I feel like it's a little bit of a cultural thing, like in the front-end community. It seems that even JavaScript in general, I feel like videos are much more common, but growing up in the C++, no one will tell you anything. Just read man pages and like, MSDN documentation. I have a hard time when I can't scroll past stuff. So video is like the antithesis for that.ADRIANA: I know, right? Yeah, that's my biggest issue also, is, like, I need to be able to skim and search easily, and video just makes it a little bit harder. I use video out of desperation. It's like I have no other options, and all Google is giving me is a bunch of videos. I'm like, crap.ORI: It yeah, exactly. But for some things, it can be useful. And I get why it's more popular with front-end, because you want to see stuff in many cases.ADRIANA: Yeah, totally. Yeah, I definitely agree. I definitely agree.ORI: So yeah, I can understand it.ADRIANA: All right, final question. What is your superpower?ORI: Yeah. So can I say my wife? Can it be not something about me, IADRIANA: That's probably our most creative answer so far.ORI: Because I guess, like, at Otterize, we're not even two years old now. And being a founder is very demanding in terms of time and stress, and even the stress is showing up here and here [points to hairline], I like to say. So my wife has been really supportive this entire time. And just beyond not just being supportive or understanding, she talks to me about stuff, and it really helps me think things through, which can be important.And honestly, I think I would probably do a much worse job if not for her, because she really helps me process things, which is really important in such a stressful environment where you can if you're too stressed out, you can reach for knee-jerk reactions. And that's the opposite of what you want to do since you want to manage and not react. So she kind of helps center things for you.ADRIANA: That's awesome.ORI: Yeah, it's honestly, it I think that the biggest thing that has had an influence on how I do things. She gives a lot of space and support.ADRIANA: That's great. I really love to hear that. And I think this is actually a perfect segue for our topic of conversation because, as you mentioned, you are a founder of Otterize. So why don't you tell us a little bit about what Otterize is all about, how the idea came about, and what it's like to be a founder because you hear stories of the stress.ORI: Yeah, I think okay, let me start with Otterize. I was going to start with the end. So, Otterize...what we do... The name is a funny joke that alludes to what we do. So we do authorization for your back end. So it's called Otterize, because if you're Israeli and you say "otterize". Authorize and otters...they're cute animals. So it's a win-win. It's a pun.ADRIANA: Totally. Totally.ORI: So what we do is we make declarative zero-trust easy. We have released open source tooling that lets you map your Kubernetes clusters. And using that map, you can then auto-generate what we call Client Intents. So each backend service in your cluster declares its intentions, which is what it needs to access. So say I have a service that needs to access a database, an AWS IAM resource, another service. So my intent would say, I need to access this thing, and it does this in a high level Kubernetes resource called Client Intent. And then Otterize figures out how to configure your infrastructure to make it work, whether that means something like Kubernetes Network Policies or AWS IAM Policies or Kafka ACL Rules. Whatever infrastructure you have and we do that all open source with a Kubernetes operator.And I guess in one sentence it would be to make access control not a nightmare because there are so many different kinds. And as a developer, what do you want to do is I just want to call this freaking API and have it work, but then all of a sudden you've got configure IAM policies and that other service that's in a different cluster, so they have a different way that you need to authenticate and authorize.So as an aside to all that, I still get to do quite a bit of hands-on work, which is fun, and it's thanks to my awesome team that is very independent and awesome.ADRIANA: It that sounds super cool. And I think as our software is only getting more and more complex, and IAM is, I think, the bane of our existence in this space, and to be able to make it accessible, easier, less nightmarish, is awesome. And definitely very much needed in this space. And to be able to also codify in a standardized way, who'd have thought? I think that's a very awesome use case. So yay. Nice to see a tool like that in the space.ORI: It you know, as a developer, you want to say, I need to connect to that thing. You don't care if it happens with IAM and you need like a thousand different permissions, so we want to take all that away. The security team cares, you don't care, so you shouldn't care. High level.ADRIANA: Is the intent that you work like...developers work with the security team to kind of determine what those policies are supposed to be, but then the developers can, I guess, self provision that access? Is that how it works?ORI: Yeah, essentially, because what we've learned is that in most cases, security teams don't actually care to approve the policy except in very sensitive cases. Like if the ledger service for a bank has an API that transfers money so that one might be approved on a case by case basis, but in a lot of other cases, they just want to know that the access is intentional, that it's not an attacker.Just like they want to know that the code that gets deployed to production has been reviewed and approved, but they don't want to review the code themselves. And developers also don't want to talk to the security team. So both sides don't want to talk. They just want to get access and for it to be intentional.ADRIANA: Yeah. That's so great. I love that use case. Going back to we understand what Otterize does. Tell us a little bit about starting up this company. How has that been, and have you guys been around?ORI: We've been around since January 2022 and, well, it's been a crazy journey, as I guess any startup founder would say that.ADRIANA: Yeah, I would imagine.ORI: I think the biggest surprise for me would be I came into this as the CTO, and the CTO at a small size manages the dev team in the early stages. So I thought like, hey, this is going to be similar to managing a small team of developers, which I've done many times before. But what it turned out to be is I found out that when you start a new company, it's not just about managing the team because in an existing culture, an existing company, a lot of decisions have been made already.When you go to make a new decision, like, should we use this language or this library? Like the smallest decisions within an existing culture. A lot of decisions around that serve effectively as guardrails and in a new company. Even though everyone we hired were people we knew personally and worked with and have even worked together sometimes, some of them when we all joined, like day one, everybody has been to different companies. Like the previous company was different. So they all come with slightly different expectations for culture, for technical choices.So the challenge is less about how do you make...Bigger companies, the challenge is about how do you move quickly and how do you keep quality and all that. And for a new startup, I feel like the challenge is how do you end up with the right culture and do it quickly while everyone gets a chance to express themselves and feels included, while taking into account that everybody has different expectations day one, and that all happens really quickly. Like at a big company, when you start a new team, people might trickle in, but you start day one with four new people. So now you're four developers and the founders, and you're like, oh my God, so all of this has to happen at once. And everybody also really wants to work quickly.ADRIANA: Right, right.ORI: And beyond that, it just teaches you a lot about how a business works, like any business, the technical stuff, how does payroll work, all kinds of regulation. You learn a ton of stuff, which is really interesting and useful.In my day-to-day, I just got a mortgage, and I understand how banks see lawyers and people and companies and how to navigate that, which has been really useful now that I'm super busy.ADRIANA: Yeah. Um, so how did it all get started?ORI: I guess I didn't answer that. So as founders, we have experienced the pain of solving for authorization and authentication at every company repeatedly. And this is like authorization is one of those things that people just sort of accept for what it is. You start a new project or you need access to something else on IAM, so you accept that it's like this. But it doesn't have to be that way because you know, Android apps, when you build an Android app or an iOS app, you know that if you specify, "I need to open the microphone," you don't have to say which specific API it is or it's very high-level. And you know that if the user and the user sees all the permissions that they're going to need, and if they approve that, then your app is going to work.And you don't have the same level of confidence and simplicity in Cloud Native. You have to say on AWS IAM, if you're using a service that is writing to like, say, if you use Cloud Watch, it's writing to S3 buckets. And you got to make sure you also have access to write to the S3 buckets that is going to access beneath it all, which is insane. You're just a developer. You're doing one simple thing, and you have to think so deep, which, I mean, I love, but I think it slows it down if you have 500 engineers and everybody needs to understand this to make progress. And different teams use different tech stacks. We've seen people struggle with it. We've struggled with it and we see how it can be better in the mobile world. So like, why not for Cloud Native? Shouldn't accept it.ADRIANA: I think that's such a great idea. I think we see this recurring theme, too, in our space, which is, like, we keep solving the same problems over and over and over. Right? And you get to the point where, like, let me just package it into a damn tool already. Right? Because it's starting to get annoying. Like, why do we need to keep reinventing the wheel? We got better things to do.So I think that makes a lot of sense. Now, you mentioned AWS with Otterize. Does it work with other cloud providers as well?ORI: Yeah, it will.ADRIANA: Okay, cool. Nice. You got to start somewhere, right?ORI: Yeah. I think the reason no one has done this before, even though for mobile apps, it seemed kind of obvious, so both Google and Apple did it is because there are so many different kinds for other kinds of software that it's hard to even put them together. I mean, I don't think a lot of people would put network policies and AWS im policies, even though they both have the same word in their name in the same category, they would think of them as different things. It's an architectural problem, right? And you can't just imagine it all together. It's exactly the kind of problem that you would start a company for if you need to work on it for a while with a team.ADRIANA: Yeah, absolutely. Now, switching gears a bit, because I think this is the thing that got us connected initially. You all are doing some cool stuff around OpenTelemetry at Otterize, so why don't you share some of that with our audience?ORI: Yeah, sure. So we have built the authorized network mapper, which prior to OpenTelemetry, the way it worked is you set it up on your cluster, it's zero config, it's all open source and it captures DNS traffic and uses that DNS traffic to create a map of your cluster. Now, with OpenTelemetry, I've actually had a chance to work with OpenTelemetry relatively recently, but alsoa while back at my previous employer when parts of it were still called open tracing. And it can be a bit of an involved deployment to get your first few bits of data because you need to integrate SDKs and all that.So the Network mapper being an infrastructure level component that you deploy in your cluster, if it was able to export OpenTelemetry metrics, then it could make the first step to OpenTelemetry adoption a lot easier. Because I think the first step when you add instrumentation is to ask yourselves, what do I want to instrument? And if you don't know what you have, which in a large enough organization as the platform team, which might be keen on implementing OpenTelemetry, that can be where you are, that you don't even know what are the different components, which is connecting to whom, what are the dependencies that I care about.And the Network Mapper had that info and the awesome team at Lightstep/ServiceNow saw the Network Mapper and contributed an awesome pull request that exports OpenTelemetry metrics from the Network Mapper. And it was really a no-brainer for us to accept the contribution and support you guys because, you know, we think of the Network Mapper as there are a bunch of cybersecurity products with simple similar capabilities for network mapping, but we really wanted the network mapper to be a standalone thing that you could use independent of the rest of Otterize.So it's really like seeing what was possible to do with Grafana Tempo so easily with the same data that the Network Mapper has. We saw how it could make people's lives easier and really, that's what the network mapper is trying to do. To be a simple tool. You can just "helm install" on your cluster with zero configuration and get as much value as you can in a complete open source fashion.And that really works well with OpenTelemetry. So far, the Otterize Network Mapper had a CLI that was using Graphviz to create a visual map of the traffic. And you could also hook it up to Otterize Cloud to get an interactive map. But now with OpenTelemetry, you can hook it up to your Grafana instance and get an interactive map, which then helps you see, okay, I'm interested in this service and it's communicating to these three other services and that's where I want to start with OpenTelemetry.ADRIANA: Right.ORI: Yeah, it's pretty powerful...the combination, I think.ADRIANA: Yeah. And it's so cool to I think it's nice to see more and more vendors, including OpenTelemetry as part of their products, because, first of all, I think it shows the staying power of OpenTelemetry, but secondly, like, the recognition that, "Hey, this could help our customers, too." Right?ORI: Exactly. And mission number one is to make people successful in what they do.ADRIANA: Yes.ORI: And we really want Intents and Otterize and easy network mapping to be something that everybody in the Cloud Native community has access to. So we have aspirations to turn Client Intents into the way that you do authorization, even independent of Otterize, contributing to upstream.So that really falls in line with that. How do we make the network mapper more useful? Like a no brainer.ADRIANA: Yeah, absolutely. So have you seen internal and external benefits as a result of the work around the network mapper, like, with OpenTelemetry?ORI: Um, so it's still quite early.ADRIANA: Fair enough.ORI: I think it's just been out for about a week or so and we are I believe we're going to publish a blog post together that will help it get noticed.ADRIANA: Yeah, definitely.ORI: So right now, I think it's depending on people organically discovering it, which, I mean, it can happen and the Network Mapper gets organic traffic, but people who are specifically looking for OpenTelemetry and using it with Grafana Tempo are going to be best served by content that points to that. We want to add a tutorial for that too,ADRIANA: Cool. And do you have any future plans around continuing OpenTelemetry integrations?ORI: So we still need to explore that. Off the top of my mind, I'm not well versed enough to say what, but definitely I think even now there's more data that the Network Mapper has access to. Like, if you have Istio or Kafka, then it can tap into resource-level or topic-level information which can probably be reflected in OpenTelemetry as well in the same infrastructure way. And once we add more capabilities to the Network Mapper, we want to add cross-cluster discovery and the ability to discover infrastructure outside of Kubernetes from within Kubernetes. So that could also be interesting to add to the OpenTelemetry support.ADRIANA: Right. Awesome. And can you speak to how OpenTelemetry is being set up or are you guys running like a Collector as part of your internal infrastructure or...what does the landscape look like?ORI: So we're we're pretty new in terms of the internal deployment. I mean, I've had the opportunity to use OpenTelemetry before, but at other eyes we really only got into it with you guys saying, "Hey, remember Grafana Tempo?" Which we knew about before, but then we were like, "Of course." So as part of dog-fooding yeah, we started with setting up a Collector and a Grafana instance, but it's also really fresh at Otterize. We make it a point to use everything we put out, including contributions, so we know how it works for users. So we're pretty early there. But yeah...ADRIANA: Yeah, amazing how's the learning curve been for you and within the company in general for OpenTelemetry.ORI: I think there are some pretty cool tutorials and guides for we've looked particularly at Grafana Tempo which we said, videos and text.ADRIANA: Yeah, I always appreciate the tutorials with the screenshots.ORI: So Grafana Tempo has tutorials with screenshots, so getting like the initial setup has been pretty easy. But we did like I think if we built the integration completely ourselves then we would have had a slightly steeper learning curve because Clay at Lightstep did the research for us of what the metric should look like and how to configure it. So it was pretty straightforward. But I think also for a user that's trying to use it now, it would also be pretty straightforward because it was just one configuration value that we needed to pass to the network mapper after all is said and done. But I think it's interesting now to explore what else can we get from it now that we have the base layer of information there is.ADRIANA: Yeah. As a follow up, do you envision at some point adding Open Telemetry traces to your core product, for example?ORI: You mean like our back-end infrastructure or like the product itself?ADRIANA: For your application code.ORI: Yeah, because we use some level of Datadog APM to debug. So there's like a natural hook point that we would go into and hook up to OpenTelemetry now that we have it set up.ADRIANA: Right.ORI: Datadog was like we set it up just because it's the natural thing we reached for from previous experience. But we're just one step away from OpenTelemetry now. From places in OpenTelemetry.ADRIANA: You've got your foot in the door now with OpenTelemetry.ORI: Yeah, it's like all the infrastructure is there, the traceability of the code is there. Like we have contexts and everything in Go so that it's easy to pass traces. So I think we're a middleware away, which is a tiny step.ADRIANA: Awesome. That's so great. That's very exciting. And I think it's, you know, the other thing that that's super important to underscore here too is the fact that you had somebody making an outside contribution to your code base and it speaks to the power of open source, really, to be able to have your...your code's out in the wild. And someone saw this and they're like, "Hey, I got a cool use case."ORI: Yeah, absolutely. First of all, it was a pretty major contribution, which was cool. It's actually the first major contribution that we've had. We've had contributions before but they were of a bit smaller scale so it was exciting on that front as well. But yeah, the cool thing is that we were actually aware of Grafana Tempo but we didn't think to do that if we had thought about it, I think we would have built it ourselves before. But it's cool that somebody saw it, saw the potential, and just wrote the code, and now it's out there, and it's open source, and anyone can use it, and you guys contributed to it.And there's actually one other company, Komodor.io, which has incorporated the Network Mapper into their own product and their own open source, which is cool.ADRIANA: Oh, cool.ORI: It's cool to see people use it in ways we haven't thought of. That really. It's like you said, it's the power of open source that we don't need to think about every use case because someone else can think about it and add it, and all we need to do is to support and go like, yeah.ADRIANA: Yeah, totally. I mean, basically you've provided the seed and then people are just like, "Hey, what can we do with this now?"ORI: Right. And it's kind of like the pitch for Google Fiber in the United States was, we don't know what people will do with one gigabit of data, but we will build it and see what they do. So the Network Mapper is really that it's just it's raw data of your network, and it does the collection bit, and then you can just add more and more exporters, which just this is like an OpenTelemetry exporter. So there's so much you can do with a map of your network. I think every developer product and every cybersecurity product has a map of your network, so it's the most valuable and flexible kind of data for a development teammate.ADRIANA: Yeah, absolutely. And I'm just spitballing here, but I think a really interesting I wonder if there could be like...a really interesting use case with...I don't know if you're aware of the Open Telemetry demo.ORI: Which one exactly? I guess not.ADRIANA: So there's a repo called Open Telemetry Demo and it's based on the Hipster Shop app, which I believe originated with Google and they've turned it into like a telescope shop now, but it basically showcases instrumentation with OpenTelemetry. So it's this multi-microservices app written in different languages and so it shows different instrumentation in different languages with OpenTelemetry. And it started last year, just before KubeCon, and it's turned into this massively complex, but very cool showcase of OpenTelemetry's capabilities.And it made me think of something where...so recently there's this company called Tracetest. So they're part of this company called Kubeshop. Tracetest is the product and they offer trace-based testing. And so basically the idea is, like, you're already emitting traces. Why don't we take those traces and create test case pieces out of these? And so they recently integrated trace-based testing into the OpenTelemetry Demo to showcase hey, like we can leverage this to basically run automated integration tests.So then I'm thinking, hey, wouldn't it be kind of cool to have Otterize, especially the Network Mapper, integrated into this demo, showcasing, again, OpenTelemetry, the Network Mapper emitting metrics. So anyway...ORI: Yeah, it it a it's a great idea. I wasn't aware of it, but we definitely should do that. And it's funny because most of our tutorials are based on the Google Microservices demo. It's the we still have the stock shop. It's not a telescope shop, but yeah, I guess that app is everywhere. I've seen it on Calico too, and Istio for sure that's a good way to help people discover that the Network Mapper can emit OpenTelemetry metrics. I'm sure it will be useful to a lot of people.ADRIANA: Exactly. And especially it's an open source tool working with another open source tool.ORI: The way God intended.ADRIANA: Yeah, that's exactly. By the way, so we will be releasing this episode during the week of KubeCon. So by the time everyone's listening watching this episode will be out on the Tuesday. Will you be a KubeCon look for the Otterize?ORI: Yes, definitely will be in the main area. Damn. I don't know what our booth number is, but...ADRIANA: Look for the otters.ORI: maybe I can check and you can...yeah...Look for the otters. Maybe I can check and you can edit that in.ADRIANA: Yeah, you know what? I can add it to the show notes once you find the booth name or in booth number, I should say. Do you guys have, like, really cool swag that you're giving out?ORI: You guys, we're going to have the booth other plushies and T shirts, of course. Of course. Oh, my God, the otters are so cute. They have, like, this little key. I wish I had one to show you now.ADRIANA: Damn it. That means you have to go to the booth.ORI: It's like it's been...yeah... You're going to be there too?ADRIANA: Yes. I'm going to be a KubeCon. I'm going to be giving a couple of talks there. There one on platform engineering, actually, and self-service tooling.ORI: Yeah? I'll come see.ADRIANA: So I feel like it's, like, very on topic.ORI: Yeah, I'll be in the crowd. I'll go, "Woo! Yeah!"ADRIANA: I'm going to now make sure that I visit the booth because I definitely want to score one of those otter plushies.ORI: And socks. Will we have socks? Actually don't know. Last time we had socks, they were a huge hit. The first time we had plushies and socks, I was kind of worried, like, would the platform engineers that come to the booth be happy to get those. Maybe it's a bit like, what do grown-ups do with plushies? But then I was like, no, it's actually a hit. And grown-ups have kids too, so not only are they interested in one plushie for themselves, they need also one for every kid so nobody gets there jealous.ADRIANA: Yes, that's right. Because if you bring a plushie home and one of them is not for your kid, you are in trouble. It is the law. I do find the socks are quite popular because I find, like, for me, T-shirts usually are very hit and miss because I'm very petite, and so if it's not very small and fitted, I swim in the T-shirt. So the socks end up working very well because you can't go wrong with socks. They generally fit.ORI: Yeah, and startup socks tend to be like happy socks, very colorful and...ADRIANA: Yes, exactly.ORI: ... maybe you don't want to walk around every day branded, but the socks are like your little secrets.ADRIANA: Yes, exactly. They're perfect. They're perfect. Well, we are just coming up on time, but before we wrap up, are there any parting words of wisdom or hot takes that you would like to share with our audience?ORI: I guess...I'll go for words of wisdom. I'll try, I'm not good on the spot without my...ADRIANA: All right.ORI: ...without my wife, but I guess the thing that I would keep in mind as a developer is to always think about the people that will encounter your code and read your code. A lot of times we think about the design, the performance, does it look good? Does it work well? But I think the most important thing to pay attention to is about the people. And sometimes the people includes you. I'm sure a lot of people watching have come up on a piece of code they wrote and think like, what the hell, what is this? And then find out it's them a couple of years ago.ADRIANA: Oh, my God. It's so true. What the hell was I thinking?ORI: It yeah, and it like, it really goes a long way because, you know, it's the, it's the feeling you get when you use a product and you expect something to be there or you expect to be able to do something, or even the unboxing experience seems like they thought about the person who's opening the product. If you come upon a confusing bit of code that does something nontrivial and you find a comment that says why it's this way and what you should do if you want to change it, like, oh, someone has thought about me.And in some languages, like Go, you can go a bit further and use the compiler to do some of that. I think if you have a function that is doing I/O and it can block, then it's good hygiene to accept a context as a parameter and allow it to be canceled. Because you're not just allowing cancellation, you're communicating, hey, this thing is going to do something that you may want to cancel, it may block. And the cool thing is the compiler then forces the caller to actually make a choice. Like they have to decide which context are they passing in, which is a bit like returning errors. You're saying with a function signature, this thing can fail, which you can't not in every language.You can do that using exceptions, Java, you can say throws and like an endless list of exceptions. But there's nothing fun, more fun than in C getting an unexpected exception.And you're like, what is even going on? How did this get here? And it's like a leaky abstraction. So, yeah, my parting words are think of the people when you write code, not about the machines, because the machines, they're going to be fine. We're going to throw a bit more CPU and memory at it. The people are more important.ADRIANA: I really love that. That those are really great words of wisdom. Well, thank you so much, Ori for geeking out with me today. And y'all do not forget to subscribe. And be sure to check out the show notes for additional resources and to connect with us and our guests on social media. Until next time...ORI: Peace out, and geek out.ADRIANA: Geeking out is hosted and produced by me, Adriana Vilella. I also compose and perform the theme music on my trusty clarinet. Geeking out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the Socials by going to bento.me/geekingout.

About our guest:Tim’s tech career spans over 25 years through various sectors. Tim’s initial journey into tech started in avionics in the US Marine Corps and then into various government contracting roles. After moving to the private sector, Tim worked both in large corporate environments and in small startups, honing his skills in systems administration, automation, architecture, and operations for large cloud-based datastores.Today, Tim leverages his years in operations, DevOps, and Site Reliability Engineering to advise and consult with the open source and cloud computing communities in his current role. Tim is also a competitive Brazilian Jiu-Jitsu practitioner. He is the 2-time American National and is the 5-time Pan American Brazilian Jiu-Jitsu champion in his division.Find our guest on:LinkedInX (Twitter)InstagramFind us on:All of our social channels are on bento.me/geekingoutAdriana’s X (Twitter)Adriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:Transcranial magnetic stimulation (TMS)Generational traumaSelective serotonin reuptake inhibitors (SSRIs)EpilepsyADHDAnxietyDepressionThe Razor’s EdgeNeurodivergentGlimmer vs TriggerBrazilian Jiu Jitsu (BJJ)Psychadelics and mental healthMental Health Resources (Canada)Mental Health Resources (USA)Transcript:ADRIANA: A note to listeners. This week on Geeking Out, we will be talking about mental health issues, including suicide and suicide ideation.Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And geeking out with me today for the second time...I am super happy to welcome back Tim Banks. Welcome back, Tim.TIM: Hey, Adriana. How's it going?ADRIANA: Not too bad. And I'm so excited that you agreed to come back for a second show because of something that you posted online recently that just like I don't know, it kind of got me, like, all verklempt thinking about, like...it was on mental health. And I'll let you open up the conversation.TIM: Sure. And I'm sure there's probably one at the beginning episode, but just the content warning, talk about things like mental health, self harm, suicidal ideation and attempts and stuff like that. So just understand that this is going to be a real and raw. I'd been working on my mental health and been talked about.I've been in therapy and stuff like that, and treatment for depression like medicine and stuff like that. And I've been open about that. But I had a couple of life events happen that kicked off a pretty bad depressive spiral that was already in the middle of a depressive episode that resulted in a suicide attempt at the beginning of July, which was obviously unsuccessful, but only barely.And I don't want to use a wake-up call because it's really more than that. It was like I really have to focus on nothing but my mental health for a while. Like nothing but my mental health. I had to do that. And it was things like having people, my network of friends inside in Tech and outside of Tech, but definitely some folks inside of Tech who, you know...I can't be alone.They were driving like half an hour to come stay at my house for the entire day or two days or three days, right, to I wasn't alone because I couldn't be alone. They were calling, they were sending stuff, they were sending food and things like that. I had my mom come into town and just help me out and just really focusing on nothing but myself, my own mental health. And even that, I mean, that was just the beginning. There was a lot that has to be done.And the reason mostly that that had to be done is because I had kicked that can down the road for so long, right? And there are things I could have been doing, should have been doing, could have been more diligent about or conversations I could have had earlier down the line that would have probably not come to this point. And they say everything happens for a reason, and I'm sure it was, but I would sure hope that I wouldn't have to endure all of this for that reason.For everyone who's listening, I am, night and day dramatically better. So I've been going through treatments called transcranial magnetic stimulation after I finished intensive outpatient DPT therapy, still on my antidepressants and just been really doing a lot of work.And I feel mentally and emotionally better than I have probably any time in my adult life. And I'm almost 50. But the hard part is this was not like a sudden thing.The very last part of it was sudden, right? But the road to get to there was long, right? And as I look back, I realized that there's, like I said, a lot of stuff I should have been doing, should have been focusing on a lot of red flags that I realized now, that I wasn't okay, right?But there's levels of marginalization that sometimes made it difficult to see or recognize or talk about. And it's funny because the one way the patriarchy screws men over specifically, is that we have this weird thing about not talking about our feelings or emotions unless they're anger, but we can't talk about being hurt. We can't talk about being sad or scared or stuff like that. We're socialized to not do that. We're socialized to, quote, unquote, be strong for the family or whatever like that. And that's bullshit.ADRIANA: Yeah, I totally agree.TIM: Can't do that.ADRIANA: Yeah. And I think it's so important for us to have these conversations out in the open because there's so many people suffering in silence on a daily basis. And it's so nice to hear that you had such a wonderful support network when things got really bad for you, that you had people who really wanted to make sure that you were okay and were taken care of, which is so nice. But I think that comes from being open about our mental health issues as well.TIM: Yeah. And I think it was interesting because I was not only open about it to the community, but also to my family, especially my children. Father of five, and I have four under 18 that stay with me from time to time. And this happened while they're with me. I've had conversations about my mental health before, but I tell them I am sick, right?ADRIANA: Yeah.TIM: I'm sick and I can't do the things that I was used to doing or that you would like me to do right now. I can't. And having those conversations when my littler ones are like...they don't really know how to categorize it. They just know that I'm not well. My older ones kind of understood, and I've talked to them more in depth about that kind of things because I want them to understand that they're probably at some point going to go through this.Mental illness can be genetic, or...however passed down it is, it can be passed down. So I don't want them to feel like I don't understand or that I don't want them to ever feel like they can't talk about these things with me, right? But also understand I'm at a very crucially low capacity, critically low capacity.So I'm going to have to ask you to self manage some things or take care of some things or talk to one of my adult friends and your adult friends that we have in your life or other folks like that to help you out. Obviously, as it's gotten better, my capacity has increased and they've noticed. But having that conversation saying, like, I understand you have these expectations, right?This is what I'm capable of delivering right now. And right now capable of delivering is near zero. And being frank about that. And that's the thing when you tell people, it's like, "Hey, I am not okay." I'm not only not okay, but I have been faking and struggling and masking for so long that I have far beyond depleted, right?ADRIANA: Yeah.TIM: And we do that, especially neurodivergent folks. We will a little be the walking wounded. We'll just be held together with bobby pins and bubble gum, just making it through there because we are so used to masking. People who suffer abuse are used to masking, right? You just cover it up and hide it over. And we should probably not do that. But also we have to give people space and safety. These are the buzzwords that actually are important.But people have to have space and safety to feel like they can talk about these things, right? That was really why I was okay talking about at work. I called a couple of friends I had from the hospital...called Kat Cosgrove, who's a good friend of mine, and I was like, "Hey dude, here's the deal..." She was absolutely wonderful.And then I just got a lot of stuff done for...in ways that I was thinking about, "I'm going to lose my job or something's going to happen." And she just mobilized in a way beyond what you would do just for a coworker, but for someone who is a friend. And I cannot be grateful enough to have her and other folks like her in my life.And so I think going out and being that vulnerable and saying to somebody like, "Hey, I really need this help. This is where I'm at and I need this help," and having that person take the ball and like, "I got it."ADRIANA: Yeah.TIM: So building the support networks are important and it's so funny. We talk about in tech, we spend a lot of time and money and have a lot of resources and tooling so we don't have to talk to each other and we should talk to each other more and not about tech. I draw my beat constantly is that tech...the actual tech is the least interesting part about our jobs, right? It is about people. It is about connections, it's about communication.ADRIANA: It is so true. And I think having those lines of communication open with our friends and family is so important. And also, I think back to my parents' generation where talking about mental health issues is so taboo. Like, "Oh, that person's crazy." They're going to be...There's so many negative connotations associated with mental health issues from that generation to the point where then people won't seek the help that they need, and then there's the tendency of passing that mentality down to your kids, so then they sort of see mental health treatment as like, "Oh, no, there's something wrong with me. I'm broken." Okay, so, yeah, you're broken. Fix it.TIM: It it the phrase you're looking for is generational trauma. And that's exactly what it is, especially among marginalized communities or immigrant communities like that. But think about it. Like in our parents' times, they didn't really have mental health care as a thing that you could walk around and have if you were crazy. You went to a place where crazy people went. That's literally how they thought about it, right? And then you were stigmatized forever after that point, right? And so we've come a long way since then as far as the resources and tools that are available to us.But we have come a long way, I mean, objectively in how we talk about it. But also, I don't know if we address it really with the seriousness that it is, because it goes to like, "Hee hee...I'm on antidepressants" as like as a stand up bit, right, to, "Oh, we should watch out for suicide."And there's a long way between those two that we should really be talking about. And so I'd mentioned to you before and mentioned other things that the treatment I underwent, transcranial magnetic stimulation, I only knew about because one of my friends in tech told me about it after.She was like, "Oh yeah, I see what you went through. This really helped me and you should look into it." Right?And that treatment has been life-changing. But I've also talked about meds. Like, I have a conversation about what meds I'm on, and people are like, "Oh, I've done this and look for this," or "This has worked for me." Other folks like, "Oh, you're on this. I should try that."Because we do that for code, right?ADRIANA: Yeah, true. So true.TIM: We do that for literally anything else. Why can't we do that for our own mental health?ADRIANA: Yeah. Absolutely. And it's really just, like, keeping that dialogue open, and I guess...being curious, too, right? Asking the questions, "What are the things that I can do to improve my mental health?" And sometimes that can be like research on your own or asking friends, like posting it out to the community...I think anything like that. To get that information right, we have to arm ourselves with information and try to squash the disinformation, because that's the other thing that will go, right?TIM: It is...it is really bad sometimes. Or the stigma is like, oh, the right wing Twitter, if you're on SSRI, is like, oh, you're on...Like half the country, I feel like is on SSRI sometimes, but it's not really that. It's not that bad. And there are more people that, you know, walking around on mental health medications than you would ever imagine or who are having things to deal with, whether it's epilepsy or whether it's ADHD or whether it's other neurological conditions.That are separate from your kind of traditional mental health things, but still things that they have to deal with that have side effects that have diminishment of their available capacity. And in the end, when we talk about it from a practical standpoint on teams, when my depression and my anxiety are really giving it to me, right, my mental and emotional capacity is exhausted, right? I don't have it. And now that I don't have it, I'm like, "Wow, I've got capacity for days." Right?And so recognizing that as a thing, the practical matters, how can we do as people in business, tech and leadership, you probably want to get a gist of how folks are doing, right?"Hey, how are you doing?""What's your capacity like?"And in a very judgment-free way, it's like, hey, man, look, you don't have to be all in depth of your employees lives and have to request information that they don't necessarily feel comfortable giving you, even if you make it a safe space, right? But you can just say, "What's your capacity?"So that way I know how to task you or give you fewer tasks (more of the point) so that you can focus on stuff. Like, "What's your capacity like?" And if it's low, what can I do to help you?ADRIANA: Yeah, absolutely.TIM: If you just ask those two questions in your 1:1s, right? That's a lot. Like, "What is your capacity?" And if it's low, what can I do to helpADRIANA: And those are definitely conversations that we don't hear enough of. And then you end up in situations where there's a lack of empathy, right, because you don't fully know, well, you know, this person isn't performing up to their full capacity because there's all this other shit going on, right? But you haven't had that conversation. So now it's just like this resentment.TIM: Like, this person should be doing more. They're not doing this or compared to doing whatever, right? And you don't know what they have going on, and they don't necessarily need to tell you, right? Obviously, if someone has a long-term low capacity thing, let's talk about this and see what it is.But there could be...maybe there are a lot of things you can do that involve, hey...that don't involve necessarily, like butting into people's mental health. Burnout is a thing, and it happens, right? And burnout will affect your capacity. So it's like you, as a leader, need to be more mindful of what your team's capacity actually is, right? Instead of what they want it to be, or what it, quote unquote, should be. Of what it actually is, right?ADRIANA: Yeah, absolutely.TIM: And this goes very much into a business and like, leadership kind of aspect, because we run our people at Razor's Edge for, quote, unquote, productivity developers, right? We are pushing for more and more and more and more and more productivity without adding the number of people, mostly, especially when they say, oh, we're in this, quote, unquote, preparing for this economic downturn, which is bullshit, because there isn't one. But but ideally they'll say, "Oh, we need to get this and we need to do this." "We need to meet these numbers," or whatever. But you, as not a manager but as a leader, need to understand what your team's actual capacity is, right? Especially you've been running over or at capacity for an extended period of time. You have to rest, you have to recover, you have to push back on things. Everything can't be the top priority, and that's what leadership is, right?And say, like, "Hey, all right, hey, we really need to ease off the throttle.""We need to come in here, get some maintenance."We need to, like, "Hey, on-calls have been really brutal." We got folks who haven't had vacation in six to eight months, right? We've got folks we need to regroup, we need to let folks rest. We need to put the yoke down for a while, right?Or just eats up...[inaudible]...and go into like, and we have this notion of velocity that's fueled by gamification and metrics that said, "Oh, if we do this fast, this number commits."We have to have more and more and more velocity and more and more and more and more, quote unquote, productivity, right?ADRIANA: And then you eventually hit a wall.TIM: And you lose empathy with it. Yeah. And you can't understand why you keep squeezing this rock and no water comes out.ADRIANA: Yeah, it's so true. And it reminds me of I remember when I was a manager, I was drawing upon...trying to draw upon some of the negative experiences that I experienced, having crappy managers who didn't care about my mental health and vowing that I wasn't going to let that happen to me. Making sure that your direct reports are okay, right? Like asking the question, "Are you okay?"Making sure that if you notice that they haven't taken any vacation for a long time, like, buddy, you need to take some vacation. You absolutely need to. And I think the other one, too, is as a manager, I think it's important to advocate for your direct reports, but also encouraging your direct reports to advocate for themselves. Because sometimes it can be so easy to just ask more and more and more of them and lose sight of the fact that you might be burning them out. And so kind of teaching them, encouraging them to call you out and say, "No, I've had enough. I can't do this anymore."I think it's really important to have those kinds of conversations, and I think it's part of making that safe space with your team as well, so that they do feel comfortable enough so that they can come to you and say, "No, I'm tapped out. I can't do any more."TIM: I think also but it's hard, especially if you're running at diminished capacity. You already have a lot in your plate to summon up, to muster up that gumption, to be like, okay, I need to push back and work. I think we should...early. But when people are in it, I'm all for like, hey, so anybody can advocate for anybody, right?ADRIANA: Yeah. Yeah.TIM: If your work spouse or your best kiki buddy over there at the water cooler or the person you bullshit with around in the group chat, whatever, if they know something's up, you can advocate for that, especially that person has more privilege, has more seniority, has more whatever, more ability to withstand any pushback, right? They'd be like, "Hey, you know what Adriana needs? She's going to need some time. You need to take it easy on her for just a little bit. Like, I'll take some things, assign some out. We can push that. That's not important. That can do it, like, things like that.And then you as a boss have to be like, word, right? And obviously, like I said, if it's for a couple of few weeks, whatever, we can figure out if it's something that's long-term, then there...you know...you can take some time off, take disability, whatever, but like, to understand that that this is not okay, we'll get to it eventually. No, this is acute. This is a need. This has now become a priority, right? When someone is having problems, that is a priority, right? Because no feature you're going to launch is going to be worth more than what could happen to people if we do not give them the time to rest, recover, and focus on their mental health, right?ADRIANA: Yeah.TIM: Somebody's like, oh, yeah, we launched this feature, but three of our developers swallowed bullets. I'm like, okay, wow, I'm so impressed with your velocity. We need to do a better job of putting the people first. And it's not just devs, it's operations folks, it's finance folks. Salespeople, I think, probably are very stress and anxiety-driven, and I think we don't appreciate that enough as sales as developers, because we get salaries, a lot of them get paid on commission. So there's a lot of anxiety around that and there's a lot of pressure on them, right? And a lot of that we can fix. Culturally, fixing the sales culture is probably never going to happen, but we could hope. But that said, you can still, especially as a leader, you can fix that culture in your company, right?You can have realistic stuff, you can have realistic expectations and also leave time for your people to be people, right? And so, I don't know. I am fortunate to be where I was. I was fortunate to be my company at Dell with the people I was working for. But if I was with there are other companies I work for, I know that I would not have had the support that I had here. And it sucks, right? Also, because I know that I've been here, I've been in this almost 50 I've been in the industry now. It's my 27th year. If someone who's junior may not have felt the ability or like, they could say, "Hey, I really need this time release," whatever.So that's why it's important that we who've been there for a while advocate for the other folks that may not be able to advocate for themselves.ADRIANA: Yeah, yeah, I totally agree, because I think back to my very first job out of school, and I worked for a consulting company, and the hours were absolutely brutal, and I hated my life. And I got to the point where I was still living with my parents, and my mom was like, every day, "Are you coming home for dinner?" I'm like, "Mom, I'll tell you when I'm coming home for dinner, because the default answer is no right now, because we're working ridiculous hours."And then I finally hit a point where I'm like, I just cannot take this anymore. And I spoke up for myself. Like, everyone else on the team was dying, but I was just like, I don't care. I'm going to speak up. And I remember getting a bit of a reprieve, but boy, did I feel guilty because I'm, like, the only person complaining about it, which is terrible.TIM: It is and also, there's that thing of, like, leadership by example. Like, if you want to have your people have healthy relationship with work, you should probably have a healthy relationship with work. And I've talked in the past about the idea of divorcing my self worth from my job, which helps a lot, because that way it's like, hey, work is work, and I can like, work, and I love my job, right?But it is not me. It's not who I am. It's just what I do for a living, right? It is not what I would do if I was rich right now, you know what I'm saying? But once you can do that, once you can compartmentalize the things that are they're part of your life, but they're not your life. Once you can compartmentalize those, it gives you clear to focus on that kind of stuff.Like, if you're worried about my career, I was like, I promise you, I don't care so much about my career as I just want to get paid. Right? But I don't have to be a superstar. I don't be a rock star. I just want to make money, right? I make the money. I don't care. You know what I mean?ADRIANA: Yeah, yeah.TIM: And so, that's helped because I know that when I have really been invested at work, like, personally in the politics and all of that is devastating for my mental health.ADRIANA: It's exhausting. Yeah.TIM: Because there's so many things outside of your control. Yeah.ADRIANA: It's so exhausting. I mean, honestly, that's one of the reasons why I went from manager back to IC. I'm like, no thanks. I just want to be off in my little corner doing my thing.TIM: It is great that people can step into those roles, because those are necessary things. But also, you still have to approach them, like I said, with a healthy relationship to work and with your job know?ADRIANA: Yeah, it's so true. It's so true. Yeah. Because you can still get wrapped up in the job regardless of what position you're in. And I can speak for myself. I have been talking about...in my circles, having a healthy having that work-life balance, which is super important to me. But I'm also sometimes the worst at it. I finish work and I can't stop thinking about it.And then something happened to me this summer where my mental health took a toll, where it just got to the point where I was having a physiological reaction to all the things that were accumulating. And then on top of that, I lost my mom last year to cancer, and it's coming up on the one-year anniversary of her death. And so all of these things accumulating, this anniversary coming up, and me just realizing that I've been hustling so hard and haven't had a chance to stop, and constantly thinking about work and body's, like, "Hey, guess what? You're done."I had to take some steps to take care of myself. I promptly found myself a therapist, which I'd been putting off for years because I had a bad relationship with a previous therapist. So that kind of scarred me like, he was an asshole. And so I didn't go to therapy for years, but I'm like, "No, I need a therapist." I need to put processes in place to basically end of the workday. You're done thinking about work.TIM: Close the laptop and done.ADRIANA: Now it's time to think about non worky things. So, yeah, I mean, putting this shit off does not do you any favors whatsoever.TIM: You know what's funny is that my ability to compartmentalize work has been greatly enhanced by working at home than by going into an office. Because when I'm commuting to work, I'm still thinking about work, right? Commuting back from work, I'm still thinking about work. I'm thinking about the whole process of getting ready to go to work. I'm thinking about work, right? But it's like when I walk into my office door here, I'm working. When I close the laptop, I'm not working. And that context switches so fast.ADRIANA: It yeah...it's funny. I don't have a problem with switching off from the working from home. Same thing. Like, just because I'm working from home doesn't mean that...(now I'm not) thinking about the work things.TIM: I look at, like, playing Call of Duty. Like, look, I have an Xbox. Call of Duty is right there. But if I'm not playing Call of Duty, I'm not thinking about Call of Duty, right? You know what I'm saying? I'm not like, OOH, Call of Duty is there to think about. I have a KitchenAid, right? I'm not thinking about baking because my KitchenAid is sitting right there. Right? Okay, cool. I'm not in the kitchen. I don't think about KitchenAid. I'm not even thinking about it. Maybe the context of what am I going to cook tonight? But I'll do that at the time of thinking about it. And so that's why I have this room set apart. Now, obviously, I have the space to have my own office, but you can have your own routine to do that. But I think what's more, the point that I'm trying to make is that one of the things that has helped me out is having these routines, right? Having these boundaries set and then being able to like, hey, this is my boundary. Like, my laptop is closed. I am done. I'm not going to look at this rest of the day.Some people don't have their stuff on their phone. So you'll have two phones, whatever it is. Right. But have boundaries around what you're going to do. And that's not just for work, obviously.That's on other stuff. There's people that like...I am this person, people are going to come to me, and I'm going to solve all the problems and do a lot of emotional labor. And it's difficult for me to say, like, "Hey, I don't have any more"...you know what I mean? I don't have any more capacity left in me. You're going to need to go to somebody else for that.I feel you. That sucks so much. I don't have it right now. I can't pour from an empty cup. And a lot of us do that, especially women with children, right? You all have to do so much. Not that men don't ever, but we all know that women, by and large, have to do a lot of the emotional labor when it comes and physical and manual labor when it comes to the household and taking care of kids and stuff like that. So you're context switching, but you don't ever really get to turn offADRIANA: Yeah, definitely. And I definitely felt that more like when my daughter was really young and I had to have to rush out of the office to pick her up from school, and I'm like, but I haven't finished solving this problem. And then it's like, well, never mind. Time to start job number two, you know? Mom mode enabled.TIM: Yeah, and it's it's harder when they're little. But also, I think one thing that helped me is that, you know, me and my ex, who we co-parent with, we've been very good about raising adults, right? The kids have responsibilities. They have stuff they got to do, like, hey, man, look, you're ten years old. You know how to empty a dishwasher. I'm not doing that.ADRIANA: Yeah, totally.TIM: Right? There you go. Have at it. You want to use the Xbox? Cool. You're going to learn to put the silverware away, and little things like that help. Because in essence, what I'm doing is I need to delegate these things to you because I don't have the capacity to do it all, at least not for a long period of time, right? And a lot of times we feel like we can't do that. Why? Because, oh, you're supposed to be able to do all this. You're supposed to do this, you're supposed to do all that. No, it's supposed to get done. I don't have to do it. Right. There are other people who consume from this who eat at the table. Cool. You can take the plates away. But it's so hard because a lot of us, especially a lot of us who are prone to tech or neurodivergent, like, if I want it done right, I have to do it myself, right? And what we can do is just like, hey, if it's put away, don't stress it, man. If the forks in the spoon thing, it's okay, I promise. If the plates are where the bowls go and the bowls where the plates go, it less than matters. It is so inconsequential, right?And letting those things go, there are some things that matters. Obviously, you don't want to put the knives pointing up. That's dangerous, right? But for other stuff, like, hey, the towels are folded in quarters instead of thirds. Like, oh, okay. Like, fine. You know, like, I don't I'm not going to stress about those. Kinds of things they're put away, right?ADRIANA: Yeah, totally. It's stuff that you don't have to do because someone else did that.TIM: What the real thing that that comes down to is getting perspective and prioritization of, like, concerns, right? Everything can't be important. If everything's a big deal, then you end up doing it all yourself, right?ADRIANA: Yeah, I so agree with you, because I am totally that person who...I will sometimes...like, my husband will refill the dishwasher and I'll rearrange his stuff, and there are times where I'm just like, no, I don't care. That's fine. He's doing it, not me. The whole burden of the world does not have to fall on me. I'm not always successful at it, but I'm trying to train myself because yeah, otherwise you'll just drive yourself mad. You don't have to be perfect at everything, and nothing has to be done your way exactly as you said, it just has to be done.TIM: Like I said, you go and have that conversation, and say, "Look, I'm at capacity, and I can't keep doing this. I'm going to need you to pick up the slack here." I'm all these things I've been doing, I'm going to need you because I know you can, right? Take a little time away from that. Do these things. I need you to pitch in, right? Because they need to get done, right? And then you're asking them to do that. And what you're going to do is not freak the fuck out if something's not exactly how you like it, because it's actually not a big deal.ADRIANA: Yeah, true. And it's about, again, asking for help, because sometimes your other half will be like, "la la la la la"TIM: Oblivious. Yeah.ADRIANA: Yeah, yeah. Right? Whether it's your partner or your kids, sometimes they're just doing their thing. You got to just like, "Can you please do the thing for me? Because I cannot right now."And you know what? They'll say yes, most of the time.TIM: Yeah. And it's like and it's like, you know, kids don't like these chores. I'm like, hey, I need you to take care of this. Or can we or can we get this taken care of? Like, it's going to take a little bit of day. It's really going to help me out because I've got this, this, and this and this. I was like, look, hey, if you want to go here and do this thing, I'll go do this thing, right? And the thing I do is like, hey, if you want to go rewrite this architecture and record this thing, that's fine. I would love for you to do that, right?And I'll take care of the dishes. I feel like there's a skill gap there, and you're fully capable of taking care of these dishes. And it's funny because the way that I reward and incentivize my kids for that and the way I extend them grace when they do things right, that's the hardest thing it is for me to do for myself, right? Extend myself grace because I didn't do this thing exactly right, but it's okay. Or to reward myself in a healthy way for like, hey, man. As a means of recognition for like, yeah, I did this thing, and it was hard, right. A thing that I've been really working on that with my therapist, I really discovered. And when she told me this, I was in tears because it was so impactful.She was like, there's this thing called a glimmer, right? And so I don't know if you've heard of this before, but a glimmer is the opposite of a trigger, right? So where triggers, they trigger like a trauma, like a trauma response or something, like negative, right? A glimmer is like a trigger, but for positive things, for self love or things you love, things that remind you of good stuff, right, or make you feel positive. So I look for like, I have triggers, but I find and make glimmers, right? Sometimes they happen. I'll give you a great example just on the driving home from TMS today, I was, like, singing a song that's a favorite of mine. And one thing TMS has done is made me a better singer and made my penmanship better. And I haven't figured out why, but it does. And so I was like, man, I'm really singing this song. Well, that's really good. I'm a pretty good singer. Little things like that and that just like, yeah, man, I get okay, that's hot. Just little things like that make an impact. So I collect those things and I write them down, right?Keep a little highlight reel or whatever. These are things I've taken the time to recognize in myself, to help myself out, right? Because I feel like we do this really well with other people. Well, to some extent, and not so well with ourselves. We as a culture I think I've said this before, even on here, we as a tech community are awful at recognizing accomplishments.ADRIANA: Oh, my God. Yes.TIM: We're terrible at it, right? Okay. A company might give you a bonus or something like that, but we'll recognize, like, hey, this person is the most impactful, blah, blah, blah, blah. Right? But it's I've been like, people who like, hey, man, they really pulled this thing off, or they were instrumental for this. Like, this thing. We don't recognize the people that chop wood and carry water that well. And we should. And I don't know how we do it, but we should.ADRIANA: Yeah, absolutely. I think we just need to get more in the habit of giving each other kudos.TIM: Yeah, this person's really...Sorry, go ahead.ADRIANA: I was just going to say, I think one of the things that we need to do is just really...in not just celebrating each other, but also ourselves. I know I get into this horrible rut of, like, I accomplished this great thing, right? Like, say I get accepted, my CFP got accepted at Blah Conference. Five minutes of celebration, and then I get depressed right away because I'm like, this is my peak.TIM: Yeah.ADIRANA: I am never going to achieve anything else beyond that. And that's, like, very self-deprecating behavior. So I like your idea of writing down your little pick-me-ups, your little glimmer points, because I think we need to get into these more positive habits. Otherwise, it's so easy to fall into a funk.TIM: It is part of the work I've done, right? And I think it helps is to really recognize, like, oh, man, this is what this means, right? I've watered my plants for, like, a week. I'm like, oh, it's not great little things and recognizing it's like anything else. You know how when we have an outage, we start figuring out how to instrument around that and we have things like, uhoh, this thing is at capacity, this thing is disk full. This thing is page errors. We can do that for ourselves. We can literally have Observability for...ADRIANA: Oh, my God. Yes.TIM: ourselves, and we probably should. Observe the human. And not even just for ourselves have other people, like, hey, my network of people, I have told them, if check in on me and if this is this mention it. Like, bring it up. Like, hey, how are things going? Right? If I haven't gone to Jiu Jitsu in a couple of days, something is wrong.And it's like things like, have, you know, keep the people around you aware of that and keep yourself around that, like logs, journals, whatever. Like, hey, I haven't it's been a few days since I did this. I'd probably need to check in.ADRIANA: Yeah. It's interesting that you mentioned if you haven't gone to Jiu Jitsu for a couple of days, time to check in, because I kind of felt the same way with...rock climbing is my happy place. I love it. It just makes the stress go away. But I had a point a couple of weeks ago, which is when all of the emotions started just crowding in, and I found, like, now, all of a sudden, my safe place, my happy place, was no longer my happy place, and it was a place of stress. And I had to walk away from my happy place for a bit because I was in mental distress. I did not feel like I was in a good place. And when your happy place is threatened like that, I feel like that is like an alarm bell. It's like screaming at you, there is a problem.TIM: Yeah. And it's it's weird because I had that with Jiu Jitsu. Whether it's gym drama or anything else like that, it's like, man, something's going on, because I'm not looking forward to this. And I do. I think that when we talk about going back, that notion of Observability and then touching back about what we talk about what leaders should be doing, there's a level of Observability you have to have into your employees as far as, like, hey, in your 1:1 check-ins, like, "Hey, how are things going? What's your capacity? Like, what can I do to help?" And if you notice that folks are running however you measure task doing or productivity, quote, unquote, or just experience, right? Keep tabs on that.And then be like, hey, I noticed that things aren't going you're kind of a little bit off, and that's fine. It's not a problem yet or anything. I wouldn't say yes, it's not a problem, but I just noticed things are different. I just want to know, see where you are, see what I can do to help, find out what you need, what I can do to help, and giving people that space to do that and also actually doing it, don't just let them say it. You actually got to do it. Helps out a lot. It makes people feel supported.ADRIANA: Yeah, absolutely. And you really nailed it. Like, just giving them space. It's not just like lip service, because I think a lot of companies will pay lip service to mental health issues, but won't actually action anything. And I think that's why it's also so important that you have companies, I think, are embracing more of this idea of taking a mental health day, and that being embraced also through managers who are like, yes, you need to take that mental health day. That is not a problem. You do what you need to do because otherwise you end up with people who are just so burnt out they can't do what they need to do.TIM: Yeah. That has long-term effects in your businesses, whether it's turnover, whether it's not, quote, unquote, productivity or velocity, but how efficient are your practices, how good is the product you're putting out versus how often are you deploying things like that? These things affect your organizational resilience. And we're still fresh off some of the impacts, the worst impacts of the pandemic. And a lot of companies realize they don't have organizational resilience, right?That people are sick, people have problems, and we don't know what to do. We don't have to handle or manage it, especially if we can't observe them from day-to-day at their desks, right? And a lot of companies punted on the difference between management and leadership, and the companies that punted on that are ones that are now we have to return to office. I'm like and if you are a leader or somewhere and you have made the decision that people have to return to office, that is for you, not for your people. Understand that a lot of companies, they work just fine, do amazing work with primarily remote workers, right? Your culture doesn't allow for that. And that's a leadership problem.ADRIANA: Yeah, absolutely. And honestly, it gets me whenever you start hearing these back to the office mandates, because we obviously proved for, what, two, three years that we did a pretty damn good job of working from home. And now it just feels like people are using the excuse of, like, oh, well, you need FaceTime or whatever, just to justify the fact that I spent millions of dollars on a new office and I look like a giant ass.TIM: Sell it.ADRIANA: Right? For real.TIM: Sell the building, rent it out. Yeah, turn it into residential, man. There's a lot of people that need homes right now, you know what I'm saying?ADRIANA: Yeah. So true. Yeah. Even in Toronto, where I'm at...mega, mega housing crisis. And yet you've got all these people, all these organizations with back to the office mandates for these downtown high rises just like, turn them into freaking condo buildings.TIM: You...there's also, like, the other congestion, like, infrastructural problems you create by having. But I think more than anything else, it's also like people who have built an environment at home that really works for them, and then you tell, hey, I need you to come back into this open office. Hard floors, LED lighting, fluorescence, like, no, I'm...not blah desks. I'm not here for that.ADRIANA: Yeah, yeah.TIM: No control over your environment. Yeah, I would rather not. And that will have an impact on people's mental health, like, for folks that are not into talking to everybody. And what I think is so funny is like, well, people got lonely. I'm like, great, those people can come back into the office.ADRIANA: Yep. Yep.TIM: I'm not saying don't have an office, but if you want to go in there, go in there, get around the other people, right? You don't have to force everybody to do it, man. I promise you, right?ADRIANA: Yeah.TIM: If people won't go into the office without you forcing them to come in the office, that should tell you something.ADRIANA: Yep. Oh, yeah. I totally agree. And then I feel bad for the poor folks who are mandated back to the office. Like, oh, you have to go back, like, three days a week, and then they go to the office to go to Zoom meetings. Great. I could have done that at home. Thanks. And now I have to dress fancy to sit at an office for a Zoom meeting.TIM: I think a long story short is that we can go a long way in kind of recognizing where people are with mental health. We go a long way to helping people out with that. But most importantly, the thing we need to do is we need to talk about it and talk about it earnestly and what's in vulnerability and talk about like, this is what works for me, this doesn't work for me, things like that. Maybe I'll have you on for a stream or something like that.One point, because I definitely am going to talk about psychedelics and mental health and how that's helped me. But people have interest in it. But do we talk about it? Because I know when I was coming up it was like, oh, if you do psychedelics, you're going to be some stoned out hippie on the streets. I'm like, I make a lot of money in my job. There are a lot of stigmas we have to break on how people medicate and self-medicate and how they deal with things. And I think that the better environment we set up for folks to have these conversations, the more illuminated we'll all get.ADRIANA: And that's why I'm so very happy that you were able to come on and share your stories and struggles around mental health. Because we need to keep these communications channels open so that more people can feel comfortable about sharing their stories, so we can help each other as a community and destigmatize this whole thing around mental health. Because healthy mind and body equals healthy human.TIM: Yeah. A rising tide lifts all ships. You create a good developer experience by giving them an experience. It has to be a holistic view on developer experience. You can give them all the tools in the world you want, but if they're sad as fuck, right, it doesn't matter.ADRIANA: Yeah. And it's reflected in your organization's outward persona, right? It really is. It's like when you bake angry, it comes out in your baking. Code angry, it comes out in your coding or whatever.TIM: Yeah, I love that because I think about my grandmother when she would argue with my granddad because my grandmother's Mexican when she would argue with my granddad and she would make salsa or chile. We knew as soon as we heard them arguing that we were going to be in for it. At dinner time, we knew we were in for it.ADRIANA: Oh, my God.TIM: Boy, she was cooking mad.ADRIANA: Damn. Well, we are coming up on time, but before we wrap up, do you have any final parting thoughts or advice or anything that you want to share with our audience?TIM: I think, honestly, talk about your mental health struggles as openly as you feel comfortable and if you don't feel comfortable, reach out to some folks who have said that they're open to talking with you. I am always open to it. I may not have the capacity at that time and I'll let you know, but I will be like, hey, yeah, let me talk about I'm happy to hear about it, right? Just talk to somebody. Start there, right. And understand that you are not alone. You don't have to deal with this alone. If your support network is not doing what it needs to do, then I don't want to say get a new support network, but widen your net, right?There are folks out there that will bear the burden with you. For sure.ADRIANA: Yeah, absolutely. Yeah. And that's really important to remember that there are good people out there who are willing to help us out, so making sure that you connect, find those people in your life.TIM: Yeah, absolutely.ADRIANA: Awesome. Well, thank you so much, Tim, for geeking out with me today yet again. Y'all don't forget to subscribe.TIM: Thank you, Adriana. I really appreciate it.ADRIANA: Oh, yeah, no problem. Y'all don't forget to subscribe. And be sure to check out the show notes for additional resources and to connect with us and our guests on social media. Until next time....TIM: Peace out and geek out.ADRIANA: Geeking out is hosted and produced by me, Adriana Vilella. I also compose and perform the theme music on my trusty clarinet. Geeking out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the Socials by going to bento.me/geekingout.If you or someone you know is struggling or in crisis, help is available. In Canada, If you're in immediate danger or need urgent medical support, call 911.If you or someone you know is thinking about suicide, call Talk Suicide Canada at 1-833-456-4566. Support is available 24 hours a day, 7 days a week.For residents of Quebec, call 1-866-277-3553 or visit suicide.ca.In the US, call or text 988 or chat 988lifeline.org . To learn how to get support for mental health, drug, and alcohol issues, visit FindSupport.gov.

About our guest:Tim’s tech career spans over 25 years through various sectors. Tim’s initial journey into tech started in avionics in the US Marine Corps and then into various government contracting roles. After moving to the private sector, Tim worked both in large corporate environments and in small startups, honing his skills in systems administration, automation, architecture, and operations for large cloud-based datastores.Today, Tim leverages his years in operations, DevOps, and Site Reliability Engineering to advise and consult with the open source and cloud computing communities in his current role. Tim is also a competitive Brazilian Jiu-Jitsu practitioner. He is the 2-time American National and is the 5-time Pan American Brazilian Jiu-Jitsu champion in his division.Find our guest on:LinkedInX (Twitter)InstagramFind us on:All of our social channels are on bento.me/geekingoutAdriana’s X (Twitter)Adriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:PerlPythonJohnny-come-latelyPerl for early webcgi-binCustomer Service Manager (CSM)Technical Account Manager (TAM)Business Analyst (BA)Clack fanHashiCorpTerraformMySQLMariaDBPuppetAWS CDKAWS CloudFormationVBasic (Visual Basic)AzureKelsey Hightower on Open Source Pitfalls and ChallengesShay Banon (Founder & CTO of Elastic)MongoDB AtlasWinAmpICQHipChatBEA WebLogicOracleAOLNoOps10x DeveloperLong Snapper (US Football)Quarterback (US Football)Punter (US Football)Transcript:ADRIANA: Hey, y'all! Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada.And geeking out with me today is Tim Banks. Welcome, Tim.TIM: Hey. How's it going, Adriana?ADRIANA: Good. And where are you calling from today, Tim?TIM: I am calling from the balmy summer capital of Austin, Texas, where it's been over 100 degrees for 40 somewhat days straight.ADRIANA: Wow, that is very hot. I am definitely like a tropical gal, but I don't know if I could do 40 degrees over 100 degrees, which I guess is like 40 degrees Celsius for that many days straight. It's a lot. Ouchy. Ouch. Well, stay cool. I hope.TIM: Yeah. Literally, right off camera, I've got my Govee stick fan. Literally, I'm touching it right now, so I have it strategically positioned.ADRIANA: That's awesome. Yeah, nothing like a good fan to help cool off. Yeah, I have one in my office in the summer because it gets like, so hot.TIM: Yeah. I face the window, which is great for the lighting, but also then it gets, like, hot in here, so...ADRIANA: I know, right? Yeah, I have the same problem. All right, well, let's get started with some lightning round questions. All right. I swear they won't hurt. Okay, question number one. Are you a lefty or a righty?TIM: I'm left-handed...ADRIANA: Yay.TIM: but because my dad taught me how to do, like, most sports things, so I throw right handed.ADRIANA: Oh, interesting. That's pretty cool. Yeah, it's funny, there's like, certain things where I think I'm a lefty as well, but I do certain things right-handed. Like, I could not even fathom using a mouse left-handed. That just feels super weird to me.TIM: Yeah, it does.ADRIANA: But I know some left-handed people who are like, yeah, I mouse left-handed.TIM: Yeah. I've switched around. I'm like, that's just weird to me. Right? Same.ADRIANA: Yeah. Also my check marks are right-handed people checkmarks. My mom, who was left-handed, did the left-handed people checkmarks, which were backwards or mirror images?TIM: Yeah, I know. Yeah. It's a pull instead of a push. I still do the push on the check mark, too, because just because.ADRIANA: Yeah, which makes sense because of being left handed, but I think it was like, that conditioning. So anywho yeah, cool. Okay, next question. iPhone or Android?TIM: Oh. iPhone. All day long. All day long. I make good tech money. I don't need a Walmart phone.ADRIANA: Love it. Okay, next one. Mac, Linux or Windows?TIM: Depends on what I'm doing. Windows is never the answer, but I have to use it if I'm gaming. Fortunately, I'm not a PC gamer. I use consoles, so pretty much Mac most of the time. But if I'm doing running a server or something like that, it's going to be Linux.ADRIANA: Fair enough, fair enough. All right, next one. Favorite programming language?TIM: PythonADRIANA: Me too. I love Python. Okay.TIM: It used to be Perl, but then Perl just gets really difficult to read after a while, and I don't have that kind of time anymore.ADRIANA: I've never done Perl, but my job before the current one, they had a lot of Perl and I'm like, damn. I'm not going to lie.TIM: Perl powered Web 1.0. Right? If we're going to be honest, all the cool stuff was done in Perl a little bit. There are some other Johnny-come-lately languages that really got into the later version of the web. But your original cgi-bin stuff, like all the cool stuff yeah. All Perl. That was all Pearl, man. You get it?ADRIANA: I rember that. I never got into Perl, but yeah, that's what I remember it from.TIM: I'm like anyone doing the complex web stuff back in the day was like, it was Perl. Yeah, it's all Pearl.ADRIANA: That's cool. Okay, next question. Dev or Ops?TIM: Oh. Ops. All day long, Ops makes things happen. Look, here's my take on it, and it's going to be spicy, and I don't care, but...ADRIANA: That's awesome.TIM: ...devs are worried about being replaced by AI. Ops is not.ADRIANA: Interesting. I feel like we need to dig into that one a little bit later.TIM: Mm hmm. Oh, we can. We can. I have opinions.ADRIANA: Very intrigued. Next one. JSON or YAML.TIM: JSON.ADRIANA: All right, next one. Do you prefer to consume contents through video or text?TIM: Depends on the content. Right?ADRIANA: Mmmm!TIM: I don't typically like video shorts, so I like little microblogs, but anything of any substance, I like in video because I throw on a video essay and just kind of do whatever. Text is fast. If I want to consume it fast, it's text. If it's going to be anything longer, it's going to be video.ADRIANA: Yeah, that makes sense. And then final question. I just added this one on, which I actually like to ask in interviews. What is your superpower?TIM: Oh, gosh, that's a hard one. So my superpower is understanding people...I think that is a good superpower...kind of digging in and getting behind the eyes a little bit on them and figuring out what's going on. That makes a lot of sense. And I feel like it fits in with the DevRel life because we have as much tech in our work as we do, like, the people side of things. Yeah. Understand. Understanding. Not like I use this, but like, okay, but what are you really trying to do? Right? What is this thing trying to make you help? What is keeping you up at night that this will keep you from keeping keep you from keeping up at night? I guess maybe.ADRIANA: Yeah, that makes a lot of sense. Absolutely. And it's funny because it makes me think of so many times at work where people will come to me and they're like, I need X, and then so I think the gut reaction for a lot of people is like, okay, let's make that happen. But one thing that I've learned over the years is like, but why do you need X? What are you actually trying to do so that you have that extra bit of context? Because maybe you don't need X, maybe you need Y.TIM: So I think the main problem that I've found in software development cycles is that we are so removed from what the customer actually needs and wants.ADRIANA: Right.TIM: Because you think about the customer, how many people that a customer talks to before you actually get to the thing where you're going to develop the thing that you think they're asking for. Right? They're going to talk to a salesperson, they're going to talk to a CSM, they're going to talk to a TAM, they're going to talk to a DevRel. Right? Maybe they talk to company leadership. Right? That gets distilled down and fed to a product manager, who's going to distill something down and feed it to a project manager, who's going to make requirements that you as a developer are going to then try to accomplish. Right. But you don't actually know what the customer is trying to do. And without that context, you're flying in the dark. I've often said that development without context is just masturbation. Right? Because you're doing the thing that you think is right. It makes me feel good, but I don't know if actually no, if it's accomplishing what the other person needs.ADRIANA: Yeah, that's absolutely true. And it's interesting because I think as much as I get annoyed by Agile, I think it tries to bridge that gap a little bit more by getting the right people talking to each other versus the old ways of Waterfall, where I worked at a bank for eleven years and the developers were so far removed from the business people. We had the business people, then we had the business BAs and then we had the technical BAs and then we had the developers and it's like, what the hell?TIM: It...honestly, though, I still feel like Agile and the Sprint system still operates without a lot of context. Right.ADRIANA: Yeah, yeah.TIM: Because we're taking most of our cues from sales, if we're being honest. Right. And that's why I think DevRel is really important because we're the ones that talk to the users and the customers and the community. Right? And so the DevRel should be an engineering role more than anything else because we should be talking directly to product. Product should be talking to us. We should be talking to the engineers. We should be connecting engineers to the people. Like, yes, I love to go up there and give. Well, I don't actually don't love giving. I actually love talking. I love going there, being on panels and stuff like that. But really, who should be at the booths is not DevRel folks. It shouldn't be salespeople. It should be the engineers who are working on the product and the product managers like that. They're the ones that should be talking to the community and to the customers and the users, not salespeople. I mean, yes, your DevRel folks also, but we're going to do that anyways. Right, but if I want to make the most out of an in-person function, I want the people who are building the things that we're selling interacting with the community as much as possible, not the people who are selling. Right? Salespeople at a conference is really not it. It's not it, you know what I'm saying?ADRIANA: Yeah. I know what you mean.TIM: When you walk down to a big city and you go past a place and people are barking at you to buy this thing from the thing that you're really not interested in, that's what that is at a conference. And it's not cool.ADRIANA: Oh my god, that's so true. It's so true. Yeah. And it's interesting because you seldom ever see at conferences...like, the engineers attending. Every so often you will get that, but especially like at a startup where everyone does everything...but at the larger companies you're absolutely right that you see mostly the DevRels or the sales folks at the booth.TIM: And so we need to fix that. Companies like, well, we can only send, like, however many people I'm like, you can send more. You just don't want to, first and foremost. And it's like, you don't need to send salespeople, right? They're going to get the leads from the thing anyways. Right.ADRIANA: Yeah, absolutelyTIM: That's all well and good, the leads that no one's going to respond to. Right. But if you really want to take advantage of being in proximity to the people that use your products or who could potentially use your products, it's the people that build it.ADRIANA: Yeah, it's so true. And it makes a deeper connection because I remember I've worked the booth before doing lhe lead-generation. Like, oh, let me scan your badge. And it's a very impersonal thing. And then when you have your badge scanned, and then you go back to the office after a conference and you're getting all this bullshit spam email, and you're like, oh, my God, just stop. It's irritating. I don't want to talk to you. I'm going to unsubscribe from your list ASAP. Like go away.TIM: Or they give you the drip campaign, which I hey, I know we just talked, like, the four or five replies. I'm like, Bro, I have not replied yet to you, so I'm not sure why you keep emailing me, but it's a sales thing. It's like this thing they teach you... oh, you just keep reaching out. Like, no, man. I don't know why this became a thing. You know who I want to talk to? I don't want to talk to those salespeople, right? If you want to intrigue me on something, let me talk to the person who built it.ADRIANA: Yeah it's so true, it's so true. Talk to the real geeks out there.TIM: I think where we come off the rails and a lot of that development, like I said, is just we operate without context or we think we did something great, this innovative thing, and then the customers are like, yeah, it's not that I want to actually use that thing. It's not that great, or what's going to make the biggest impact. I would have much rather you did this, right? Because we prioritize a feature based on sometimes a specific sale or two or three customers, right? Because it's big numbers right away for these two or three customers. But you could have much bigger numbers if you met the needs of a lot of customers with a feature or with a fix, and then they grow in your product. Right. I don't need to sell a million dollars next quarter. Or what I need to do is I need to sell $20 million over the next three quarters.ADRIANA: I remember I've worked with previous jobs with vendors where they're obviously a small startup and we're obviously their big client. And so they are bending over backwards to get your business and putting certain feature requests on hold for your feature request because you are the big screaming client, which ends up being a detriment to...TIM: Always, always.ADRIANA: the company I think, as a whole.TIM: Because you have that one customer you cannot lose.ADRIANA: Right, yeah, exactly. TIM: And that is a growing pain, because you have now stopped being a company and now you're a contractor.ADRIANA: Yeah.TIM: You're pretty much held hostage by this one client and you have to bend to their will which is a very uncomfortable place to be in. There is this kind of bravery, I would say, almost, that comes...it's actually called leadership...that comes with saying, like, hey, no, this is the vision. We're going to actually develop the product to meet the needs of these customers. If you want it, great. You can buy it. That's awesome. If you don't like it, I want to know why you don't like it, right? And then maybe we can fix it. But also, I'm not going to bend over backwards just for you to the detriment of everything else. Right?ADRIANA: Yeah, absolutely, absolutely.TIM: And so I think the weird part is that people are like, there's a thing that you have to pursue this right now. We need this right now. We read this right now and a lot of people are playing checkers and we need to be playing chess.ADRIANA: Yeah. I love the prop.TIM: I don't know if this is going to be video or audio or not.ADRIANA: It's going to be both, actually So the people that watch this will enjoy the clack fan. I am so happy you brought out the fan actually.TIM: I actually have one that says shade on it, but I left that in the other room. So you just get the rainbow fan. But the point still stands.ADRIANA: Absolutely. But it's interesting on that same point. I think it also brings to mind this whole mentality, especially in corporations. And I think you see this the larger the corporation, where people stop having honest conversations with each other. You're so hell bent on not disrupting the hierarchy and like, oh you went over someone's head. Or people are afraid of owning up to their mistakes because of whatever, makes it very hard to be productive.TIM: There is this...almost politician-like cop-ing out on accountability.ADRIANA: Yes.TIM: No one ever wants to admit a mistake, or no one really wants to admit the true reason behind something, which is just, I want to make money, right?ADRIANA: Exactly.TIM: Let's take HashiCorp, for example, right? They come in here with this, oh, we're protecting open source. And by doing this and people doing this and not contributing...like, no, bro, you had an awful quarter and you're trying to make investors happy. You can just...just say that I would have much more appreciated that than feeding me copious amounts of bullshit about the open source community that you just hung out to dry, right? And in the end, it's going to cost you in the long run. I don't know about a career limiting decision, obviously, but it's certainly going to be a scope-of-influence-limiting decision for HashiCorp over the long run. Right? But they were thinking about this quarter and next quarter, right?ADRIANA: Yeah.TIM: And I'm sure it's going to make some investors are going to be like, okay, great, they're doing something great, but it's not going to be good in the long run. The community is already looking at this and going, all right, well, it's a mistake to rely on a single company for this, so let's either look for an open foundation or do we even need this at all?ADRIANA: Yeah.TIM: Think about it. When Terraform first came out, there wasn't really a good, strong API for interacting with a lot of the ways we consume hardware and infrastructure resources, right? So Terraform is very useful to have something, especially as an alternate to Puppet, right, because it was open source and much more usable and much more elastic, I guess, in those ways, keeping state and being able to compare things like that, committing things to GitHub, etc. etc., right? That's not the case anymore, right? You can have programmatic access to hardware and infrastructure provisioning APIs and health checks, and state have whether you've got CDK and CloudFormation AWS, Google's got whatever it's got. I'm sure Azure's got some VBasic scripts or something like that you can run. I'm kidding. I'm just throwing shade, Azure.But the thing you really have to ask is now, do we need to have another entire product for infrastructure-as-code, which is what it comes down to, or can we now use the native tools and native APIs we have and store state somewhere else? The answer is yes, there are open source alternatives to doing exactly that. And we were never really forced to examine our need for it until Terraform said, ha, we're going to change the license. Right? And then now people can just decide to make you irrelevant, which never would have happened if they had done this. People were just happy on like, okay, cool, we'll keep using Terraform because there's no reason to examine it. Now they have.ADRIANA: Yeah, it's true. And it's interesting because Hashi fans are like, there are hardcore Hashi fans out there, and I do feel like it alienates a lot of them.TIM: It does. And I think the interesting part is that the people who are Hashi fans now, where is your allegiance? Is your allegiance to the company or the product?ADRIANA: Yes, yes...TIM: And those are two different things now, right?ADRIANA: Absolutely.TIM: Or they were two different things and now they're the same.ADRIANA: Yeah, it definitely brings that more to the forefront, as a result.TIM: The...you know, you got a lot of discourse, you know, especially people like Kelsey who are saying, really talking about what does it mean to have open source maintenance, or how do companies, large corporations with capitalistic interests, what influence do they really have on these things? Or how should they be allowed to participate in these open source projects in ways that don't hurt the community, right? And so you really do have to kind of ask it's like, what do we as a community of practitioners and users and influencers and decision makers and things like that, what are we going to do to really change this going forward? Right? And we do have the ability to influence that. We can change that. It's just a matter of are we going to, we're going to dedicate the time and energy to it or are we going to abstract it away like we tend to want to do with everything else?ADRIANA: Yeah, absolutely, or it could be that someone gets pissed off enough, and then... what was the...I want to say there was, like, MySQL that they closed sourced, and then they forked it off to an open source version. Am I getting that right?TIM: Yeah, that was right. Oracle got it and did all that. And then well, I think first it was like when MariaDB first came out, right? Same thing.ADRIANA: YeahTIM: It was just a fork of MySQL, right?ADRIANA: YeahTIM: You look at the same thing with Java.ADRIANA: Yes!TIM: You know, when...whenever a company's been like, okay, cool, we're going to commercialize this thing, and the open source companies be like, I don't think so. I don't think we're going to do that. Before this, it was with Elastic and Amazon with...ADRIANA: Oh yeah, that's right!TIM: ...with OpenSearch, which I really think... now, I'm going to be honest, I know the inside because I used to work for Elastic and I've heard Shay talk at great length sometimes about kind of what his view on that was. And companies owning an open source product is an oxymoron, right? Open source, the community owns it, right.ADRIANA: Yeah, absolutely.TIM: And if you want to dip into that well to make some money, that's fine. Everyone else can too, right? If a company owns it, it's not open source.ADRIANA: That's true.TIM: I mean, they have the ultimate control over it in the end, right?ADRIANA: Yeah.TIM: And we're seeing that over and over again. It's like companies, instead of going through and developing products and service around products and services around this product that make people want to spend money on it, right. They're seeing competition getting upset, and they're trying to stifle competition rather than just getting better at it.ADRIANA: Yeah, yeah, that's true. It's interesting because it makes me think of OpenTelemetry, which is open source, and before, in the Observability world, everyone had their own tracing implementation. Right? And then OpenTelemetry is like, no, let's standardize it. Right?TIM: Yeah.ADRIANA: The cool thing about it is all of the major Observability vendors are fully supporting it, and so you have the community coming together to support this thing, and so what differentiates the vendors is not their proprietary SDKs. It's, how do they render the data? So it changes the conversation, which I think is, like, really cool.TIM: And that's the way it should be. Right? I want to choose a company that offers this service based on the quality of the service, right? Whether it's faster, whether it's more reliable, whether I get better support if I've got these people that do, like, hey, I used to work as a MongoDB, as a service provider, right? And we competed directly with MongoDB Atlas. We competed with AWS to some extent, things like that. I was like, we were just better at it. That was all it was. We were just better at it. Right? And that was fine.But the other thing, too, is..."Better at it." Does that mean that we have enough? Are we essentially like a lifestyle business, or are we trying to make billions of coin, the market, blah, blah, blah? And so I think that's, too, it's like, what is your end goal? Like, there's always this thing. Well, you have to be the biggest and baddest thing. It's like that late stage capitalism thing. No, actually, I'm cool with only owning, like, 20% of the market I'm cool with only owning 10% of the market as long as me and all my employees get paid, right? And customers happy and we do kind of organic growth, that's fine. I don't understand whatever happened to just kind of growing normally and organically? That being okay. Right? We make a profit, people get paid. People can go on vacations. Everyone should be...I don't understand why that was never good enough, right. Or was good enough. And now it's not.But what you end up happening is you have a crush and a grind to get some stuff out, some innovation happens, and as soon as people start trying to return money to investors or go to IPO, now it's the same standard playbook like everyone does. Okay, great. Now we're going to do this, and we're going to do this. We're going to do this. And you see it over and over again. And so all the promise that you see behind a company or a product that gets you excited, you already know that the shoe is going to drop and get short-lived, and now they're going to add like, they're going to ruin it.ADRIANA: Yeah...TIM: Do you remember WinAmp?ADRIANA: Yeah, I do!TIM: Right? WinAmp was great, right? WinAmp, I think it was, like, 2.54. It was like, the thing. It was so amazing.ADRIANA: Yeah.TIM: And do you know what happened to WinAmp?ADRIANA: I don't...no...TIM: They got bought by AOL.ADRIANA: Oh...TIM: and AOL ruined WinAmp.ADRIANA: Oh, jeez.TIM: Yeah, it was awful. I will never forgive them for that. They ruined WinAmp.ADRIANA: Oh, jeez.TIM: Uh, do you remember ICQ?ADRIANA: Yeah, absolutely.TIM: ICQ was great. It is still, to this day, my favorite messaging platform. You know what happened to ICQ?ADRIANA: I don't, no.TIM: They got bought by AOLADRIANA: Get out of town. I did not know that.TIM: Yeah, AOL ruined ICQ. Right?ADRIANA: Oh, man.TIM: Do we see this trend, like, what's happening? A pattern. When we talk about this thing of, we're going to buy this thing and then stops. Do you remember HipChat?ADRIANA: Yeah, I vaguely remember it. I never used it, but I do remember it. Yeah.TIM: And it was kind of cool.ADRIANA: I never use it, but I do remember it. Yeah.TIM: It was kind of fun. And then it got bought. I think it was either Slack bought it from Atlassian and then shut it down.ADRIANA: Interesting. Oh, shoot.TIM: And so when we start seeing this kind of thing where we're like, oh, well, we don't actually want to build a better product, right? We want to eliminate people's ability to choose between products, right? So we don't want to actually have to compete because that's too expensive, right? So it's just cheaper to buy them out. And then not have to make something better, right?ADRIANA: Yeah.TIM: And so that kind of thing is like, I don't like that. Let the community decide if your product is better. Great. People will use it. They'll buy it, right? Or if there's more of a compelling reason for them to use it. If it's less expensive, if it...service is better if it's just more available, whatever it is, right? Let people choose that, right?ADRIANA: Yeah, absolutely.TIM: Don't kill the competition with a pen and lawyers. Kill the competition by building a better product.ADRIANA: Yeah, I totally agree. I totally agree. Yeah. It makes me think of Oracle as well. I spent many years as a Java developer, and so in my Java days, like, the Java app server, my favorite one was, like, BEA WebLogic, and I think Oracle bought them in and was like,TIM: But so many things that we enjoyed have been ruined by acquisition, right?ADRIANA: Yeah, it's super sad.TIM: So when we see this trend continuing through stuff that was used and or beloved by the open source community, it's just more things like, we can't trust corporate interests to have our best interests in mind, because they don't. What's good for the corporation is almost never good for the customer.ADRIANA: Oh, definitely not. Definitely not. Yeah, I feel the same way. I think that's one of the things that frustrated me when I started working was realizing, like, the world is more nefarious than I would like it to be. I'm like, Why can't we all just be friends?TIM: But I think what's worse than that is that there's also the narrative by some straight white dudes on Twitter who are like, oh, well, you're a highly paid capitalist person. You can't be against capitalism. I'm like, just because I detest the system doesn't mean that I'm not good at it, right? Matter of fact, because I'm good at it is why I detest it, right? I'm coming from a place where I know how it works and I know how to make it work for me, and it's terrible.ADRIANA: YeahTIM: I shouldn't have to, right? That's not hypocrisy, that's understanding.ADRIANA: True. Yeah, I totally agree with you.TIM: But I'm not going to mention any names, who it is. They know who they are, and they're probably listening to this.ADRIANA: Now, pivoting...I want to pivot back to a thing that you said earlier when we were talking, when we were doing the rapid fire questions, the Dev versus Ops. You said you had opinions, so I want to hear them.TIM: Yeah. So here's the thing, right? There has been this long, probably a good ten years worth, if not longer... But I know for ten years at least, move to minimize the role of Ops in what we do for software and product delivery, right?ADRIANA: Yeah.TIM: There's DevOps. And they're wrong by thinking this, but a lot of folks think like, oh, it's when dev people do Ops or Ops things gets automated, right? That's not what DevOps is, right? Remember the NoOps push for a while?ADRIANA: Oh, God.TIM: Like, oh, we don't need Ops.ADRIANA: It's like no code. Like, yeah, right.TIM: Yeah. What they're saying is we're abstracting the work of operations away so that devs don't have to be concerned with it, so that they can just do on development. And a lot of developers think that the sun rises and sets by whatever they type out of their fingers and it doesn't, right? The realty is that developers are a skill, don't get me wrong. It's skilled and very difficult. Not very difficult because it's not like any of us are doing brain surgery, right? It is a skilled and it is a highly complex skill set, right? But it is only a piece in the cog, right?And developers are not rock stars, right? They're not. I don't care. The notion of a 10x developer is bullshit, right? You can have all the developers in the world you want to, and you're just going to be developing...you're going to be running stuff on your laptop if it's not for operations, right? Everything happens because of operations, right? And we rely on each other. Like operations need developers to write software for them, they need developers to write drivers, they need developers to write firmware. Like all that kind of has stuff, like APIs, like that also has to work.But when you need stuff to work and you want to make money, it's your operations and support teams that actually do that, right? Developers will get you sales, operations will get you renewals.ADRIANA: Yeah, that's actually a really good point.TIM: And what does any business want more than sales? They want renewals. Let's talk about get renewals from Ops, right? But I think, I think where we come where we come off the rails in a lot of this is like, what are our roles? We're developer relations roles, right? I work for a hardware manufacturer. We don't you know, I should be talking to operations, right? If if you do any kind of API for hardware, stuff like that, you should be talking to operations folks. Unless you're selling specific developer tools, then you should be talking to devs. If you're selling things that people consume in their operations, especially anything around infrastructure or networking, you should probably talk to operations too, right? But we focus on developers, especially when it comes to salaries and stuff like that, because we're thinking, oh, well, operations can be obfuscated away and do this and this and this and this, right?But the operations knowledge gets more and more complex, like how many Ops folks when people talk about using Kubernetes as we abstract away more stuff or like platform engineers like that, they're just operations folks, right?ADRIANA: Yeah, absolutely.TIM: We're providing a platform for developers to use because we do not want them interacting with the hardware, you know?ADRIANA: Yeah, yeah. Sorry. Go ahead.TIM: And so, and as it so there's this...they change the name for operations over and over again, right?ADRIANA: So true.TIM: Because they don't want to just come out and say like, hey, these are Ops folks.ADRIANA: Yeah. I completely agree with you. And it makes me think of something. And to your point, operations folks have been tasked with learning more and more and more stuff, right? Like, picking up managing Kubernetes clusters, and learning these IAC tools and like, you know...TIM: Managing Docker repositories and then CI/CD tools and stuff like that. People who are called DevOps engineers, they're just Ops folks, but they're working on the internal tooling, right? That's all it is, right? But it's funny because really, ideally in the pie in the sky world, the very last concern that a dev is going to have is like, okay, I am now going to package this thing up. That's where it should end. And maybe not even that. I'm just going to commit it to main, right? I'm going to commit it to the production repo. That's where I want you to stop, right? Cool. We've got it the rest of the way, right?Once it gets committed to production, to the production repo, everything else is operations at that point, right? And that's kind of how it should be. I don't want the developers to have to work and not that they can't. I just don't want them to have to I want you to just go back to developing, fixing bugs, developing features, right? Operations has the rest.ADRIANA: Yeah. Now, do you think, though, that it's helpful to have that mutual empathy for what each of these roles entails? Right? Because I do feel like maybe there is this us versus them mentality with Devs versus Ops, and I've been on both sides of it, and I feel like especially when it comes to developers are done developing their feature, throw it over the wall, it's gone into production. That's it. And now there is a bug that arises, like, not my problem.TIM: So I think that's the whole thing that DevOps is supposed to be solving, right?ADRIANA: Yeah, supposed to be...TIM: The throat of the wall, the silent isolation it's supposed to be solving, like they turn into all these other things, and it's not.ADRIANA: Yeah, absolutely.TIM: DevOps is a culture, right? A culture of collaboration and interoperation, right? Where Ops has these levels, concerns and the roles and things they got to do great. It should be informed by what devs do. Devs should be informed by what Ops do. Ops should be able to say like, hey, these are things we found has this problem, this problem, this problem, this problem. So you need to work that into your dev cycle to get these things fixed because these are not operations issues. These are like software bugs or whatever. Right? Cool. That's great. Dev should be able to say like, hey, it's supposed to do this and this and this and this. So when you're designing architecture reviews like that, you have to make sure this and this, it is going to rely heavy on this caching thing. So you have to make sure it's robust. Those are the things that Ops folks should be concerned about with that. Right?I think what ends up happening, and this is just my observation as an Ops person, that Dev has very little knowledge or concern about what happens in Ops. Right. But Ops, by the nature of the role, has to be concerned about the code that's coming out that they have to deploy. Right? And I think to the extent where Devs do not concern themselves with Ops and they think that the whole world revolves around them, that poisons the culture.ADRIANA: Yeah. Absolutely.TIM: And then you end up developing kind of practices around that notion where Dev is the most important thing. Right, but the customers don't interact with the Devs.ADRIANA: No, they don't.TIM: Customers interact with Ops.ADRIANA: Yep, yep.TIM: And that goes back to what I said before about customer-driven development and having context for the things you do.ADRIANA: Yeah, absolutely.TIM: Right, but in the end, I think what's so interesting about that is that we have to have this kind of cultural coming to Jesus moment. But also that's an organizational thing because not all organizations are like that, but a lot of them have come that way and that's where the leadership comes in. Right, but we're not doing leadership, we're doing management. Those aren't the same thing.ADRIANA: And that's where I think having those honest conversations that so many organizations lack becomes a problem as well. Right? You've got people not wanting to admit when there's a problem, people being basically creating their own kingdoms and wanting to protect their own kingdoms and not thinking outside the kingdom. You launch into those sociotechnical issues that plague so many organizations. I definitely see that a lot in big organizations, but it doesn't mean that the small organizations aren't immune to them either. I mean, that can easily happen.TIM: I think that's kind of what you talk about. It's like you talk about how do we do these things? We've created so much tooling and so many products so that we don't actually have to talk one with another, right?ADRIANA: Yes.TIM: So that we try to automate, not even automate empathy. We're just kind of like, oh, we don't need emotions or empathy about this because it's not about the people. So it's like Slack. Why do we have Slack? Because we don't really want to talk to people, we want to be able to chat with them. Right? Why do we have GitHub issues where we interact with people, like over text? But a lot of times too, if you just...not that you can't do this in remote culture, but just like we're doing like, hey man, let's pop into a Zoom meeting for five minutes and we'll talk about this thing. Great. All done. Right.ADRIANA: Yeah.TIM: And yes, I understand the large distributed teams where asynchronous communication is very important, you have to be able to communicate that way, to collaborate. And that's fine. But you understand it's going to be slower, right.ADRIANA: Absolutely.TIM: But if you want to quickly, let's have a conversation. We can understand each other, have some and then I can understand your side better. Or what? Your...not even your side. I can understand what your concerns are better. Right? And have context for why you're saying the things that you're saying or why you have the concerns that you have. And now they can become my concerns, or I can address the concerns that you have a little bit better.ADRIANA: Yeah.TIM: Right, but we have so many tools so that we don't have to do that's.ADRIANA: So true. And I think part of it, too, is people are afraid of confrontation. Like, it's easy to hide behind a chat window, right? But as soon as you're face-to-face with someone, it's it's easier to hate someone or what they're doing behind the chat window. And then the face-to-face is like, there's a person.TIM: Well, I think it's weird because some people rely on other people. Like, that the stereotype of that one asshole engineer that always makes it hard for everybody else. It's a stereotype because it was very often true in a lot of orgs, right? They're relying on getting their way because people don't want to have arguments or confrontation or whatever. Right? And it doesn't have to be about that. It's like, hey, man, you actually don't have that much power. You can argue about all you want to, but once we go for this peer review...in that way, it's good.Right, but at the same time, it's like, if you have to do that just to deal with this one person, how about we just get rid of that one person who doesn't know how to be empathetic and work on a team? Doesn't matter how good he codes, if we have to implement all these things just for this one person.ADRIANA: Absolutely.TIM: You should have organizational fixes, but also you can get rid of toxic people in your org.ADRIANA: Yeah. And that's an interesting point because I think there's this mentality in a lot of teams where, like, oh, but this person is a superstar. We can't possibly get rid of them, but they're fucking with the team's mojo, and that can be worse than having this superstar who knows their shit and fixes things and blah, blah, blah. Because now you're affecting the morale of your other folks, and then that can create tension in-fighting whatever. Right?TIM: Yeah. And we see this like...I'm going to draw a parallel to sports teams. You've seen sports teams that have one superstar player and the rest team is garbage. Right? Or even if they're not garbage, that one superstar player is a problem and it causes discord on the team. Right? They're a problem in the locker room...ADRIANA: AbsolueltyTIM: ...they're a problem with the whole thing. They get so much attention. Negative attention. Right? That the rest of the team falls apart.ADRIANA: Yeah.TIM: And as soon as they get rid of that one person, they're great. Right? But you can have a superstar player that makes other people around them better. Right? You look at like, Michael Jordan, right?ADRIANA: Absolutely.TIM: One of the great examples. Was he a jerk? Yeah. Was he good? Yeah. But did he make other people around him better? Absolutely. Right. And like, that's what you want. You don't want to have somebody who's just, well, this person knows this one technology really great. You know what, I can deal without that. I don't have to have that. If you have to have that, that's a business failureship. That's a leadership failure. Failureship. Failureship.ADRIANA: Yeah.TIM: That's a leadership failure.ADRIANA: I absolutely agree. I think it all boils down to just focusing on the human aspect of...we are humans working with other humans. And we should not forget that. Like, we do not work with robots. Not yet. Maybe someday. Sorry.TIM: So we work on robots, right?ADRIANA: We work on robots. Exactly. Perfect. Yeah.TIM: But technology is inherently still about people. People consume it. People need it. They have needs that they want to address, right? We're basically using these things to solve problems, right? But again, technology for the sake of technology is just masturbation, right? We need to be able to have interoperations of team. We are software engineering teams. We are operations teams. Company is a team, not a family, right? We're a team. And so we have to interoperate.And it's so funny because if you think I'm going to go back to a sports reference for those, you know, it's like everyone on the a lot of people will make fun of the kicker until that kicker needs to save the game, right? And then all of a sudden the kicker is the hero, right?ADRIANA: Yeah.TIM: There are no unimportant parts of a team, right?ADRIANA: Yeah...TIM: People like, "Who's the long snapper?" People know who the star quarterback is, but they don't know who the long snapper is. But the long snapper can lose a game for you. You know what I mean? That's an important thing to remember. The team exists with all the people in it. It's not all about one person. Everybody on that team is necessary and importantADRIANA: Yeah.TIM: whether or not they are rock stars. And then you can extend that beyond just a single team and talk about the interoperability of various teams in an organization like you're. We're about to have some fucking tea right here, my friend. Do you know how sick I am of mostly developers I'm going to go back to that again who are very self-righteous and very degrading to people who are quote unquote, non tech roles like HR, like sales, like customer success.All these people like that. They are why you have a job because of them. I may not agree with having sales pressures, but I will never say we don't need salespeople, right? Because somebody's got to talk to these people because you can't you're not good at that.ADRIANA: That is so true.TIM: That's not your role, right?ADRIANA: So true.TIM: You need to be understand some of these pressures. You need to be able to interface with people of various roles and really go out there and put yourself on the line, right? You have to be able to make sure that we all get paid.You have to make sure that we all have insurance and make sure that we get people in here hired. You have to make sure that your video conferencing stuff works, that you're laptop works.Like all these things are not we don't have anybody that we're not going to say not anybody. For the most part, people are working there because their role is necessary, right?ADRIANA: Yep, absolutely. Things don't happen by magic and fairies.TIM: When people say, oh, the Rockstar, you know what, I've had Rockstar office managers that they paid six figures and absolutely they were worth every penny because they made the stuff happen, right? Yeah, absolutely. I've had Rockstar recruiters everybody at a company. Their role should be important and should be necessary, right? Even if they're the long snapper or the punter.ADRIANA: The yeah, totally. I love that so much. Well, we are coming up on time, but before we finish off and you've given us so many awesome hot takes, but do you have any final hot takes to impart or pieces of advice?TIM: Honestly, the final hot take and piece of advice that I want to impart is to stop having sales-driven development, right? We talked about resume-driven development. We make things overly complex and complicated just because because so and so...and we shouldn't have that but we also shouldn't have development based around a sales quota or based around anything like that. Like, no, man, make the product. Make the product good.ADRIANA: Yeah.TIM: Make the product something that people want to use. Right. And then we'll get it sold. Right.ADRIANA: Totally.TIM: But don't go chasing waterfalls. Just stick to the rivers and the lakes that you're used to.ADRIANA: I love it. I love it. Thank you, Tim, so much for geeking out with me today. And y'all, don't forget to subscribe.TIM: My pleasure.ADRIANA: And be sure to check out the show notes for additional resources and to connect with us and our guests on social media. Until next time...TIM: Peace out and geek out.ADRIANA: Geeking out is hosted and produced by me, Adriana Vilella. I also compose and perform the theme music on my trusty clarinet. Geeking out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the Socials by going to bento.me/geekingout.

About our guest:Rob Golabek is Chief Architect & CEO at Translucent Computing. A thought leader and insightful tech visionary with over 20 years of experience, Rob is a Cloud Native expert specializing in App Modernization. Leveraging data, AI & cloud for digital transformation, he provides expert guidance to clients navigating the complex, ever-changing cloud-native landscape.Rob has shaped the technology landscape through his work at Translucent, progressing from software development to architecture and leadership roles. His expertise in cloud-native technologies, DevOps practices, infrastructure tooling, and tailored consulting approach helps clients drive toward cloud-native success, including observability and robust cloud foundation building.Rob also leads the ExecutiveEspresso Series, where he contributes to fueling business growth and inspiring the next generation of innovation.Find our guest on:LinkedInX (Twitter)Find us on:All of our social channels are on bento.me/geekingoutAdriana’s X (Twitter)Adriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:Translucent ComputingEtobicokeLefty DayJavaSickKids HospitalMetavante/GHRCanadian Army ReservePinky and the BrainPoint Click CareExecutive EspressoCloud NativeAngularTekStackFalcoKeyCloakPostgreSQLRedisOpenSearchKafkaPrometheusLokiLogz.ioKratixNatural Language Understanding (NLU)Neuro-Linguistic Programming (NLP)Natural Language Generation (NLG)Large Language Model (LLM)AI OpsDALL-EcapybaraJava enterprise serverTucowsNomadVMWare TanzuOpenStackAzure StackBank of MontrealCanadian TireHashiCorpVault (HashiCorp)Consul (HashiCorp)Jaeger (tracing tool)KubernetesTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, Reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And geeking out with me today is my good friend Robert Golabek. Welcome, Rob!ROB: Hey, nice to be here.ADRIANA: Super nice to have you on. And full disclosure, Rob and I have known each other for a really long time, like since, what...2000...I want to say 2005? It's been a while. We've known each other for a really long time in a past life, in our past lives as Java developers, which is really awesome. So Rob, for starters, where are you calling from?ROB: I am from the deep west Toronto Etobicoke.ADRIANA: Yay. Fellow Canadians.ROB: Yeah, people don't know Etobicoke is a borough of Toronto, so some people call it Toronto, some people don't. For some people, I heard it's really far. For me, it's actually the perfect balance. Twenty minutes from Toronto. But yeah, get some kind of space. So yeah, that's kind of where I'm from.ADRIANA: Cool. Awesome. Awesome. All right, so we're going to start with some rapid fire questions. Are you ready? I promise it won't hurt. All right, number one, are you a lefty or a righty?ROB: Righty. And happy Lefty Day. I saw that post, so yes, your superpower...I was going to respond post of my right-handed rights.ADRIANA: I always forget to acknowledge Left-Handed Day. And then this year I'm like, "I am going to schedule this post so I don't forget." And then when it popped up the next day, like on Monday when I was back at the office, I'm like, "Oh, Lefty Day passed. Oh, I remembered post on that."ROB: It so the reason it's close to me is my dad is left-handed, right?ADRIANA: Awesome.ROB: And for some weird reason, it was weird when he was growing up to be left-handed. So they tried to even make him write it with the right hand, and it was kind of, you know, so yeah, it's dear to me.ADRIANA: Yeah, totally. Yeah, my mom too, she was left-handed and she was subjected to people trying to make her write with her right hand. And she was one of those non-functioning-with-her-right-hand lefties...everything with the left hand. So she's like, "No." I can manage with some right handed stuff, but lefty and proud. All right, next question. iPhone or Android?ROB: AndroidADRIANA: All right. Mac, Linux, or Windows for development?ROB: Windows.ADRIANA: Awesome. Favorite programming language?ROB: The one I know, I got to say JavaADRIANA: All right. Dev or Ops?ROB: DevOpsADRIANA: Yeah, I've gotten a few of those answers before. It's very PC. DevOps. All right. JSON or YAML?ROB: Depends on the situation.ADRIANA: All right, fair enough.ROB: All right.ADRIANA: Fair enough. All right. And then final question: do you prefer to consume content through video or text?ROB: Text.ADRIANA: All right. Yeah, the text people are winning so far. Most people are like, "Text." I'm right there with you.ROB: And if there was a video, I watch it on mute. I like the writing.ADRIANA: Do you read the subtitles?ROB: Yeah...I don't know, I'm not a video person.ADRIANA: Yeah, I know, right?ROB: So kind of my age, I guess.ADRIANA: My daughter Hannah, she's like video, no question about it. I'm like, really?ROB: Yeah, I'm the same way.ADRIANA: But yeah, maybe it is an age thing. I don't mind...I'll watch video with subtitles or I will just put on the audio and walk around the house and have it on YouTube...video on my phone, walk around the house with just the audio, and that I can consume...but I can't just sit there and watch a video. Especially for tech stuff.ROB: Yeah, my my attention span is like, really, like, short. I want to kind of go to the end ofthe video, and I just want to read it very quickly because I usually skim through it and then I read the most interesting part in video. It's like, okay, where's the climax? You can't really find it.ADRIANA: Yeah, I'm exactly the same, so I totally feel you. All right, so now that we're warmed up, let's geek out on some stuff. So I guess first things first. So why don't you share with everyone what you do? Because you've come, I guess, a long way from our early days in our earlier careers of being the lowly Java devs.ROB: So yeah, so maybe start from the beginning, you know? '96, '97, '95. I don't know, just kind of coding. And at that point was involved with wires and illegal streaming. Got me interested. Kind of made some money from there. Very quickly. Went to Sheridan in 2000. Within a year, kind of graduated and then got my first job at SickKids Hospital as a reports developer that turned into Java developer, that kind of turned into architecture. It was pretty cool. And after that, that's when I went to Metavante, or is that the right name? I don't know. I think...ADRIANA: I don't know what they're called anymore because it was like when I joined, it was called GHR, and then Metavante ate them up and then I don't know what happened after that.ROB: Yeah. So I don't know. In between that, I was kind of in the Canadian Army Reserves too. So kind of got some discipline there. So yeah, it's kind of put me straight as an arrow. Got me kind of healthy and got some responsibility skills. That was from like '99 to 2004 while I was at SickKids. And then between '99 and 2010 while I was still working, I had a side business. So funny story, is in my first resume that I submitted to, I had like, a quote where I want to have a worldwide business where I kind of want to dominate and kind of provide value to people. And when I gave the resume to the SickKids people, they laughed. Right. And I'm like, was that naïve or was that aspiration to kind of something greater. So the entrepreneurship was always there, looking backwards. Maybe a little naïve but kind of inspiring to something greater was kind of my goal. That was kind of my beginnings of trying to take over the world. Pinky and the Brain 2006 that's kind of where I met you in Metavante. Worked there for three years. Went to Point Click Care. I don't know, I think everybody kind of knows here in Canada. Point Click Care one of the kind of unicorns in healthcare.So I was there for six months. Sad story is I joined, somebody got the bonus for referring me - it was my brother - who's kind of with the company as me and then six months later I leftADRIANA: Right when he got his bonus I'll bet.ROB: And they changed the rule after me. I think they even call it the Rob Rule that referral...you got to work there a little bit longer. I didn't do it on purpose. That's kind of when I started my business after Point Click Care. Got my first contract kind of working and actually was with SickKids too, developing their platform and that's kind of where my journey started. And we're here today. And what we do is right now we matured and kind of through the innovation that we do and putting engineering before sales, which I don't always advise because if you have passion for engineering and you want to do everything right, it might hurt sales. But we're proud of that. We run the business our way. So because of that we always kind of innovate, not always to the benefit of kind of sales, but it got us to the journey of early adaptation of Docker, Kubernetes, Cloud Native always early adapters and now we're Cloud Native experts specializing in app modernization, trying to kind of build for the Cloud and the beauty of Cloud Native and optimization, which I love, is it's ever changing, right? So before was moving to the Cloud was legacy software. Now it's kind of the hot take is how do you add AI to software that already kind of are out there, right? In a few years it's going to be something else. So really love what I do, kind of giving the Cloud Native expertise and kind of sharing my wisdom with people.And through that, sorry, I started Executive Espresso series where I started kind of like, you know what I love kind of talking to people. So started posting information, just kind of sharing on Cloud Native expertise and kind of the different aspects...Kubernetes, Observability...one thing that's challenging, which I tell you and it hurts me, is to be Cloud Native expert. I keep reminding myself how big the space is and like DevOps, Observability, Platform Engineering and cloud foundations, it takes a lot of learning and knowing and talking to people like you and different spaces. So I find that really challenging. But I enjoy that because in my DNA it's kind of learning. So combining all those things is pretty cool.ADRIANA: And you touched on something really important, which is like the Cloud Native space is ginormous and technology is ginormous and there's a new thing out all the time, so then you can't stay on top of everything. So how do you pick what you focus on as a result of that?ROB: So we can bring up maybe when you're doing the edits, you can bring up the landscape of the Cloud Native landscape. And I don't know how many tools they have now. Maybe 200, 300, a lot. So what we focus on is opinated experience technologies that we use. So we call it our Tek Stack, kind of powered by open source software. And we chose some tools right as the starting point. Now when we go to clients and kind of try to kind of give our opinions, it's based on that. Now it's also being open to other tools. But when you choose a tool, let it be mature, let it be kind of used by people, let it be a supporting community.We did a mistake before in the past, where we were too early of an adapter and you pay the price. I think we did it with Angular 2. We did it way too fast. When Angular kind of went through versions of one to two was Angular 1, then it was 2, then it jumped all the way to 5, was too early. I wish we waited a little bit and kind of used it maybe a little bit later. And same with these tools.So we broke it down into different tools for security. It's Falco, Consul, Vault, KeyCloak, kind of maybe HashiCorp kind of world. And then for kind of cluster resources, Postgres, Redis, OpenSearch, Kafka.So you can see it's like main kind of tools that we kind of use.And Observability: Prometheus stack...Kubernetes Prometheus stack, Sentry, Jaeger, Loki...Kind of making sure that we center on those tools and then making sure that adding principal infrastructure as code kind of on top of that and on top of Google, that's kind of how we chose the tools.And that's like the starting point, right. You can see for Observability, I think it's a very similar stack as Logz.io uses or anybody kind of those seem to be the main kind of open source tools that are out there, and there's a lot of support for them. So that's kind of the biggest kind of aspect of selecting them. And they're really good, right?So, yeah, that's kind of how we use them. But the biggest thing is through clients and through conversations, you always learn about the new tools. So best way is to throw your tools out there and then tell you some.The conference you went to, I think, was from you. You threw one tool, I forget the name of it, that I never knew about. And I was like, okay, it was for platform engineering, I think, or I can't remember which tool it was. But you were using your presentation. It'll come to me.ADRIANA: Oh, yeah. Kratix.ROB: Kratix, right, right. I never knew that tool before because I never came across it. Right. But then you use that and then it kind of opens up, and then I can query you and be like, hey, how do you use it? Where's the support? So learning from the community and kind of expanding it and then making a selection, hey, is the tool that we want to use or not? Do we want to add it to our stack? Right. So that's pretty cool.And I'll finish with this. That's kind of where the new thing of platform engineering is, I think. How can we best, to your question, select the best tools that maybe if I was going to propose to you, but also switch the game, what are you comfortable with and building around that most important part?ADRIANA: Yeah, that's so true. It's so important because there's nothing worse...and I've been in the consultancy space before...and there's nothing worse than coming in and saying your stuff sucks and then you're just going to hurt their feelings. It's like basically saying you have an ugly baby and no one wants to hear that they have an ugly baby. You got to be gentle and understand. What are you comfortable with, what are you using? Hey, would you be open to switching over to this? If you're familiar with this, maybe this might be the thing for you. And I think that's very important, especially in consultancy, because you're essentially trying to help companies do things better. But there can be a lot of resistance to change, so you have to be very gentle with them.ROB: It yeah, I don't know if I'm an engineer anymore. I'm ex engineer. I love engineering, but I spend more time doing non-engineering stuff. But there's one thing, right, that I always notice with engineers kind of myself, too, not excluding myself. There's that ego, right? I selected, I know the tools better. Prove me wrong. Why are you using this tool? And I don't like taking that conversation there. I'd rather being like, hey, if it's tools great, let's use it, let's improve it. Let's build what you guys need.Right? But engineers are smart people. I'm going to say that they're usually intellectually smart, so they know what they're talking about. And you got to come with a game, too, to say that you know what you're talking about. So that's kind of where the conversation goes.ADRIANA: Yes, absolutely. Is definitely a fine line. And I think one of the things in engineering, that engineering is an art form, really. And I think that goes to say for any type of art form is that sometimes we tend to fall in love with our code, with our technology, with the things that we create. But the best thing that we can do for our art is to give it some sort of a seed so that it can grow, whether it's like, hey, that sparks another idea where someone's like, hey, you know what? You could do this a little bit better. I like where you started, but I think this is how it can be improved. And being able to let go of your initial notions and be open-minded to other ideas, other ways of improving it, honestly, I think that's what open source is all about and I think that's what makes also for very successful organizations and very successful teams that you have to check your ego at the door.It's hard though, because sometimes you're working on a thing and it's like it's your baby. You've put a lot of TLC into it only to have someone say, well, I found a better way of doing it. And it pretty much scraps all the stuff that you did that can hurt. But also recognizing that maybe your initial work, even though it's being discarded, inspired somebody to come up with a better way of doing things.ROB: Yeah. I was going to ask you, what do you think is the best way of judging that? Right. How do you best put it out there? You kind of answered, I guess, open source. Right. Kind of let the community play with it. Any other kind of ways you would kind of try it out. Kind of let kind of people give you opinions in a non hateful control fashion.ADRIANA: Yeah, generally just having conversations. I think it all comes back to community, whether it's putting it out there through open source or writing about it in a blog post or having a conversation with somebody. Finding ways to make those connections, I think is probably the best way but you can't do that without it being out there in some form or another, I think.ROB: Yeah, I really like kind of in my recent time writing, right. So got me thinking and expressing and talking to people. Right. And then the biggest thing is taking that feedback in a positive way.ADRIANA: Yeah.ROB: First reactions like, oh, man, why did he say it that way? But then it's like, why did he say it that way? Maybe explore that a little bit more. Right. And then you meet the person, and then you have a different kind of perspective, and then you can change or you don't have to change. The biggest thing is to agree with themADRIANA: Yeah, absolutely. But I think the most important thing is that someone offering an opinion forces you to take a step back and rethink it. And it's like what you said, I'll either agree or, hey, there's something to that statement. Maybe I'll tweak it or take some of that into consideration, or like, no, I actually think my way is the better way. I've given it some thought and that's perfectly all right.ROB: Yeah. And you might have different motives too, right. It could be business case, could be technology, could be different case. Just recently, we had somebody come in and they had an objective of looking from this kind of zoom...was monetary zoom. And it's like that's one way of looking at it. Right. And because it's a business client, they're going to push it in that way. Now, as an engineer, the most frustrating part is let go of your best practices. And then because most of the times client is right. Quote. You try to kind of make them happy but you got to really put your ego away and also put away I told you so because I believe even in that scenario business person could be right because now they're coming from their perspective with and they might have limits.So you got to look at from that angle ego from technology, from business and kind of move the conversation forward.ADRIANA: That makes a lot of sense. I think at the end of the day, you just have to be open-minded. So of all the technologies that you've been working with, what's the one that's really exciting you right now?ROB: I love Conversational AIADRIANA: Oh, yeah. Cool.ROB: So...and applying it to any domain. We're just working actually with Pat, working on Conversational Kube Bot, where you can talk to it in human language and get a response.ADRIANA: Oh, nice. Is that something you guys are developing?ROB: Yeah, we're developing it. I want to release it. It's kind of started as a kind of small project because we're in a grander schemes working on Enterprise search, and we call it Conversational Enterprise Search, and we call it, like, Next Knowledge Base Economy, where knowledge is king. And how can you take that knowledge and how can you converse with it right. At a basic level. Right. And applying it in Kubot, hey, get all the resources, all the material from kind of Kube Bot and then suck it in. Use kind of NLU, NLP, NLG, kind of all the kind of natural processing, human and language kind of processing. So you're able to create something where it's your human assistant. Right. So my goal is, like, I never want to remember a Kubernetes command. And with this, we already have a prototype where it's like, hey, tell me the status of the system and let's see all the pods or something, right?ADRIANA: Oh, my God, that's so cool. I cannot tell you how many times, if I'm away from Kubernetes for a while, I have to Google this stuff. Or now I have a GitHub repo where I just have a README with all of my go to Kubernetes commands because I forget that stuff, especially the gnarly ones. Like, how do you freaking go through your logs in Kubernetes? Or how do you log into your pod? Into your container in your pod? Or be like, yeah...ROB: I'm with you, man. I have a folder with documents, and it's for, like, Kubernetes, Docker this, and I'm like, get lost in those. And then it's like searching through those commands. So we're applying this Enterprise search and conversational search to Kubernetes and Observability. And it ties into AI Ops. So I'm in awe in how powerful large language models are and applied in the right case, I'm going to write a blog. It's on my to-do. I kind of have a draft format where take it from a different angle, right.ADRIANA: Yeah, yeah.ROB: How these AI tools can help the world, right. I see too much boom and doom, kind of, hey, they're going to break this, break that. It's going to take cover, control. Yes, everything, right? We have the biggest case of nuclear power. It's for good, it's for bad. It's our human choice to use it for the good. Right. And I'm always optimist.So I love it because it can apply to so many...We just combined the few elements, AI search and Observability and Kubernetes and boom. That's something we're working on.So that goes back to engineering and working cool stuff. So that's kind of what I really enjoy.ADRIANA: That is super cool. Yeah, it's funny because I think AI has definitely become a hot topic because it's come up more than once in this podcast. I think my first dabbling into AI was, like, using DALL-E for generating images for my presentations. That was kind of my first one where I'm like, oh, my God, this is the coolest thing. I can tell it to generate pictures of llamas doing funny things. What? Or my favorite, I have this love for capybaras now because Instagram one day decided to serve me pictures and videos of capybaras.And I'm like, oh, my God, this is such a glorious animal, you know, DALL-E has generated me a bunch of images of these things for my presentations as well. So I'm like, "Shit, that is some really rad stuff."And then further leveraging ChatGPT for even certain things, where you find yourself in a position where I need to reword this thing. My brain is fried. "ChatGPT, just take the sentence that I wrote and make it a little bit shorter," because I don't have the brain power to try to think of five different ways of saying this word and conveying this thing, right?ROB: So you're touching on something pretty cool, right? So it takes you to the next level. And some people say it actually does it for you. It doesn't yeah, it's going to be mind-blowing. It doesn't, because I can smell, like when a marketing person talks about technology thing, and it kind of doesn't make sense. And then when a techie will use the same kind of and they will just rephrase it. There's a difference.So it's such a helpful...I love it. It's been changing. So we applied it kind of all over the place. Again, combining AI, Observability, DevOps like...crazy.ADRIANA: It's going to be mind-blowing. And I think people forget that it's not like AI, as you said, AI is not going to do all the work for you. You still need the human touch to guide it in the direction, and then you still have to vet it because sometimes AI spits out some dumb-ass shit and you're like, "No, I do not want this." And then you just rephrase the question.At first when I heard the term "prompt engineer", I'm like, "Ha ha. That's so hokey."But we've been prompt engineers for a while now, if you think about it, in software, because that is essentially what we do when we do a Google search, especially when we're trying to solve a gnarly-ass problem and you enter a particular search term, and then you're like refining, refining, refining, until you're like, oh, you know what? That's not even the right question that I have to ask, but now I've got enough information that I know the right question to ask, and that's essentially what a prompt engineer does. It's just now the floodgates have opened in terms of what it provides you right. It's more than just those Google search results. It's more contextual information. ROB: So, you know, I agree with you. 100%. So I didn't know what prompt engineering was. That I was doing prompt engineering, right? Before it was...because I was doing what you said. It was kind of like the engineering brains, like, okay, I'm going to do it this way. I want to ask it that way. Oh, it's pretty cool. And then you start learning from it and then yeah, you were engineering a prompt, right? As a CEO, write me an email on this promotion.ADRIANA: Make it sound more beautiful. Another thing that I want to ask is, we both came about in technology before there was such a thing as Cloud, Kubernetes...We are children of the monolithic era of Java enterprise servers, which are no longer I don't know if I miss it or if I'm glad that that stuff's gone. What was your foray into Kubernetes? What led you in that path?ROB: So I was doing consulting in Montreal, this is...whenever Docker 1.1 came out and was lucky enough that the company was kind of looking and really trying to find solutions around Docker. And we use Docker Compose and Docker Compose is kind of limiting solution.And from there, just bringing the Docker world, we kind of started working with it. We had a few implementation of Docker Compose for clients, and then Kubernetes came, right? Early adapters...and kind of jumped on that because there was a limitation of controlling and deploying Dockers without a kind of orchestration platform. So we kind of started building for Google and Kubernetes and creating our own kind of platform with the CLI on how to deploy, ordering...So we were kind of early kind of working on it and the tools that we have right now weren't there. We kind of build them ourselves. So that's how we jumped on on it. So it was through a client and then just thinking that it's really cool how you can kind of abstract to OS level virtualize, a little kind of component.It was just kind of groundbreaking even though Linux had it before putting it in kind of element where, hey, we're using Eclipse. And then instead of deploying MySQL on my Windows box at the time we deployed it in Kubernetes...sorry...in Docker, which you can kind of start and turn on and off.It wasn't some kind of heavy windows or Mac installation.That kind of it was just bring it up, the Docker is there, and connect to it. And I was like, man, that's pretty cool, right? And I'm like, man, I don't know. As an engineer, it was kind of groundbreaking tech porn.ADRIANA: Yeah, I totally agree.ROB: I'm like, oh, my God, what can you do? And then not a lot of people were working on it, but we had some solutions, and then Kubernetes was the next kind of level.ADRIANA: Yeah.ROB: Kubernetes is complex, right. It's not easy. I wouldn't say for everybody to use it. There's a good case for it, but those benefits that it brought were pretty cool in terms of kind of working with containers and providing the networking and deploying. So kind of building around that. That's kind of our first foray to it. And it just continues until now.ADRIANA: I think that's such a really good point on the containerization is the gateway drug, right to Kubernetes. I mean, it really is. Docker in itself was awesome. And then you're like, oh, shit. Now I've got to manage these Docker containers in tandem and figure out all this stuff, the networking and stuff between them. And then Docker Compose kind of helps you with that, and you're like, okay, that's better. And then you realize I need a little more, umph.Oh, Kubernetes is like, the next natural evolution of it, where you're like, oh, my God, this makes things so much easier. But then at the same time, it's like, my life is hell. It's like, you can't win, right? It solves a problem, but then it brings on additional complexity because it is such a complex tool. But so cool.ROB: Yeah, it I keep following kind of...some questions. Once use Kubernetes, and people are against it and big projects, small projects, I have a simple answer. The community of tools is so big right now, you got to use it because everybody's kind of working towards one goal, and that's the beauty of it, right? Yes. It's complex. Yes. It's hard. Yes. You got to have that's what we try to make it easier. Yes. You got to remember that managed Kubernetes is a little bit easier, but dealing with it overall, it brings complexities. But having every single tool, like Cloud Native tool, you go into a landscape, every single tool is deployable on Kubernetes, right?So having that power and building from infrastructure-as-code and kind of Helm Chart and combining it all together, the power is there. That's kind of what I think is the biggest benefit. So, yeah, use it and then use it smartly. If somebody asks you when to use it or if it's good or bad, man, that's the wrong question. You find a problem, and there's solutions for it.And if you want to build a WordPress site, build it on Wordpress.org or something, right? Or if you want to deploy WordPress and Kubernetes, deploy in Kubernetes. What is your need? What is your problem?ADRIANA: I totally agree. And it's funny because I was having a similar discussion with folks today where I was chatting about Kubernetes and Nomad and how a lot of people talk about it in terms of a versus thing. But it's like, what is your use, case? When I worked at Tucows, it was a Nomad shop.And it made sense because they had their own data centers, which meant that when they tried to start up their own Kubernetes clusters in their own data center, that's like you are creating your clusters from scratch, which is a horrible, horrible experience. Versus if they were using Public Cloud and have access to managed Kubernetes, maybe that would have changed the conversation.But at the time, using data centers well, between running Nomad in a data center versus running Kubernetes in a data center, it's a lot easier to manage a Nomad cluster compared to a Kubernetes cluster. But then also, I guess some organizations might not need the additional complexity that you get with Kubernetes, and so they might choose Nomad or whatever other product because there's like, for example, VMware Tanzu, right?They're a competitor as well in the space. I've not played with it, I've just heard of them, and that is the extent of my knowledge. But it's interesting to know that there are other competitors in the space that solve the problem, but in a different manner. And maybe that suits your use case better.ROB: Yeah. So, when we were working, like, a few years ago, I felt it was nightmare to have Kubernetes on premises or data center, to your point, right?No matter something, the tools were not baked in. Now it's easier. But that one leads me to a question for you. What do you think of...I read some articles that were kind of, I guess, headline grabbers. Cloud is dead. People are going back on premises or data centers.How do you feel where the world is going to go? Like, having a crystal ball...Cloud versus maybe people going back to their own data centers or hybrid. Any ideas there?ADRIANA: Yeah, I think it's going to be a hybrid thing because here's my take on Cloud. I think Cloud abstracts a lot of the complexity that you would have for managing your own data center. And I think to a certain extent you can even manage the complexity of running your own data center through tools like OpenStack. And I think Azure has a thing called Azure Stack, and I'm sure the other Cloud providers have their own thing as well.So you're basically having the same nice little infrastructure-as-code convenience in your data center rather than hosting in Public Cloud. Now. I think a lot of people treated like there was this mega rush to public cloud, I think because A, it was easy, and B, there was a lot of hype.And then people forgot to look at the cost, where they're like, oh, this stuff is limitless. No, until you get your first cloud bill and you're like, "Shit, that was a massive cloud build."Did I actually need all that stuff? But in terms of leaving it to somebody else to manage your infrastructure, awesome. But you have to be super mindful of your costs. Whereas when you're running your own data center, you are so mindful of your costs because you are keeping an eye on that budget like a hawk. Right? It's like, no, I do not have extra rack. Like, I ran into an issue when I worked at Bank of Montreal where we were setting up...we had to buy new physical server.There was no rack space. They had to buy a rack. And because there was no rack space, they had to lay in the electrical work to be able to rack up that server. There was all this stuff that you take for granted when you're, when you're running in Public Cloud.ROB: Yeah, I think it's going to be both. I'm just a proponent that it's really hard for data centers, specifically the smaller ones replicate the security, right? How do you do that, right? So you have these billion dollar clouds and their day in, day out as they go to an office and they think how to make it better, how to make it better. And over there where yes, now there's great tools from a lot of clouds to have infrastructure kind of as code infrastructures, cloud foundations for your data center. But that investment, continuous investment into securing it, that's what worries me, right?Maybe like to your point, I'm hearing horror stories with managed services and cloud and cloud bills that it might be more kind of cost efficient to have that data center. Right. Because the cloud costs are so huge. Right. I think we're kind of still in the early stages, but I think it's going to be hybrid. I just don't know how cool will solve that issue of having a secure data center. So is it going to be your own data center stack?So maybe know Bank of Montreal or the big banks or Canadian Tire might have their own because they have money for it. Right. And then you might have data centers that are kind of from the old age where they host stuff for you and you just have your rack in there. We might solve the cost savings issue, but if we don't, we're going to see some bigger blowback. But I just don't see yet how other companies can replicate that heavy investment those big three are doing into that security or whatever security or the future tools or that's kind of where the word is going to be. So I'm going to see where it's gonna go.ADRIANA: Yeah, it'll be interesting and in the same way way that you're kind of keeping an eye on the whole data center situation. On prem or Cloud. I think we'll see a similar movement with the monolith versus the microservice, because, again, a lot of organizations rushed into the microservices model thinking, this is going to solve my problems. And then now they're rethinking it, which is rather interesting, which I'm not surprised by, because, again, it's like a lot of hype and a lot of people just did very knee-jerk reactions, rather than, is this actually going to do the thing that it's supposed to do for me?ROB: I think it is going to go somewhere in between. As long as you work towards the scalability and elastic nature of the cloud, build it for that, right? So microservices are good for that if built well because you can isolate the problem, right?If you can have a monolith, make sure you can do the same thing. Make sure you can scale the biggest thing of monolith. Once it couldn't scale and you had your 30 different features in one set, and then what, right. So there's room for both. And it's an architectural pattern they want to use.I agree. But it's the same answer. The Kubernetes answers. It's the same kind of answers. When I see kind of, hey, I go to Reddit and I'm Kubernetes and I'm here and I'm there, it's the same thing. Oh, I would never use monolith. I'm like, man, that's not the right answer. Be more a little bit critical of what you're trying to say, why you fail with your problem. It's not like brass stroke for everything is the same. So room for both.Got a question for you. Can I ask? Okay. Observability...what do you think of same thing with the cost because you're in the space and I think we had a conversation on it before, manage versus kind of in your stock because just an example, DataDogs and all that stuff, the same thing. You start slowly and then there's a boom, a bill, right? Is that bill justifiable millions of dollars? Where do you stand in that Observability world and what do you think about open source or in your kind of open source entire Kubernetes or kind of powered by open source versus kind of the fully managed solutions and the benefits kind of of that. Where do you stand with that?ADRIANA: Yeah. So I'm going to put on my not a "I work for an Observability company hat," but my "I was in the position of managing an Observability team hat," and from that perspective...so when I worked at Tucows, I came in to manage two teams, a platform team and an Observability team. And the Observability team at the time, their function was basically managing tools and not focusing on practices. But we were also using a SaaS vendor. So internally managed tools plus SaaS vendor. I'm like, you know what, you've already got the contract with the SaaS vendor. Let's use that as the standard. Let's ditch the internal tools so then we can focus on practices and focus then on making sure that people are doing Observability properly and making sure that we standardize on the OpenTelemetry.Because this was like the early days of OpenTelemetry, so traces weren't even general availability. Now we're at the point where traces are general availability, metrics are general availability and I think logs are stable, but depending on the language, it's like the specification is stable, but it's on a per language basis, like where things are. But long story short, OpenTelemetry has evolved a lot and for me it was more important coming into that team making sure that the organization was doing Observability properly rather than focusing on maintaining tools. Because if you're so focused on maintaining tools, then what's to say that you're actually doing Observability properly? So we wanted to set out a set of best practices across the org.Now, we did run into cost overruns with the vendor that we were using, but the nice thing about using OpenTelemetry is it gave us this opportunity to...because my focus was, let's make sure that the organization instruments everything in OpenTelemetry. And they were not. They were using vendor SDKs at the time. But my goal was let's inform people on making sure that they adopt OpenTelemetry so that if you're stuck with a vendor that way you're not stuck with a vendor that's going to cost you a gajillion dollars.Right now you have that flexibility of going to another SaaS vendor or...you know what, now you have the flexibility too. If you want to go the self-hosted model you have that kind of flexibility. But yeah, I feel like when you're evaluating vendor, you have to know what you're getting in bed with. Because as soon as with that particular vendor, we started moving away from their SDKs and started using OpenTelemetry, the cost shot up because they supported OpenTelemetry, but they treated the OpenTelemetry stuff as like extra I don't know, extra nodes or whatever, extra containers or some I forget what it was, but our costs shot up. It was shockingly horrifyingly expensive as a result.So I think you need to understand the cost model up front. Unfortunately a lot of vendors have very complex costing models which then that makes it a little bit tricky. Yeah.ROB: So when you said that if you design it properly, do you think you can very easily exchange the tools because your best practices are kind of build on OpenTelemetry and then you can kind of go from tool to tool? Is that kind of what you mean by best practices?ADRIANA: Yeah, so best practices means...because the idea of Observability is your system is emitting enough information so that even without knowing the inner workings of the system, you have enough information so you can tell what's happening, right? So yeah, you can use OpenTelemetry but if your system is not emitting the right stuff then so what, right?And it's a combination of emitting the right stuff and also making sure that the vendor is representing the information. So then when you instrument using OpenTelemetry the thing that differentiates the vendors is how they render that information. Is this going to be useful to you? So it's a combination of making sure that the code is instrumented properly and also is this thing showing up in a way that's useful to you so that you can troubleshoot. Right, so that I think becomes the trick.ROB: Yeah, that's good, right. It's kind of like what we're concentrating with kind of our stack. But the journey is not understood, right? And I feel some vendors are overselling the promise because the tool will not solve everything and you can just get into a really bad practice of paying a lot because you're going to be searching for what to collect and just scraping everything possible. So that best practice we're talking about and then emitting the data, collecting data. That's a very important piece.So back to the other question. So we have the practices and kind of OpenTelemetry and kind of instrumenting the code. Where do you find then after that's done, the SaaS model vendors and I don't want to pick on DataDog, there's a few others of them that are there. Where do you feel they fit into that once you have that set up the internal platform versus external SaaS model.ADRIANA: In terms of what specifically?ROB: For Observability. So comparing having Prometheus stacking your Kubernetes versus maybe connecting to again Logs.io, just say, right? Because they're kind of API based and kind of instrumenting kind of thing. Where do you think do you have an approach or preference towards one or you think it depends on the situational and company?ADRIANA: Yeah, I think at the end of the day it just depends on your situation. When I started my Observability journey, my dream was to have a tool that took care of all the things. So in my ideal world you could do away with Prometheus because you can emit those Prometheus style metrics and then just ingest them into whatever system and you'll have a place that displays your metrics, your logs and your traces and they're all correlated nicely.ROB: Right.ADRIANA: I don't think that any one vendor does that well right now it's interesting too, like for example in OpenTelemetry there's a way right now to correlate your traces and your logs which is currently being implemented. There's a way to correlate your traces to your metrics. It's called a trace exemplar. But when you look under the covers...so a lot of people talk about trace exemplars. You look under the covers. It's not been implemented for a lot of languages. I think the only one that's actually been implemented for is Java. So then you'll see actually a lot of vendors that will do that correlation in the tool itself and not use OpenTelemetry for it, which is quite interesting. So there's still some work to be done. It'll be interesting to see where things go.ROB: That's an interesting problem that I feel we always face because they're so wide to kind of adapt to so many different languages and tools and stuff and open it up and making sure can one company be doing everything well? It goes back to kind of can Apple do everything well? Can Microsoft do everything well? At what point can you invest in everything, right?So that's going to be interesting to see when I was talking to somebody at a conference, what's going to happen eventually is people are going to be really buying out each other, right? We're going to reach that level where they're going to be eating up and then, hey, these guys are doing good. This level Observability, combine it together and then see if that works.I spoke to you about it as well, kind of because where you are so that's going to happen. That's good and bad. Because that will kind of go to your point where maybe somebody's going to be able to create that kind of one tool by waiting to see if there's going to be enough appetite and investment to make those different parts of the tool well structured. So it's pretty cool. Pretty cool.This whole Observability is just so crazy, so vast. You can spend just like and you can spend a world and all your time reading about it and you still can kind of tackle the fraction of it, right?ADRIANA: Oh, yeah, absolutely. I know. I do this for a living, and I'm like, I've barely scratched the surface. Well, cool. We are just coming up on time. So, for parting words, do you have any awesome advice that you want to share with our lovely audience?ROB: Go slow, talk to experts. If you do things, try to do them right the first time, but don't be afraid to fail. And iterate, right? So it's kind of challenging aspect there, but yeah, maybe for people that are starting out, touch technology, it's here with us. For AI, embrace it, don't hate it. It's here with us. There's ways of things, figuring it out. As long as we have a positive outlook for what we want to do, we're humans are very smart, we're going to solve it. So that's kind of the approach I take too. All these different things that are coming out, maybe because we're techies, we enjoy it more because we see the potential of it and I see huge potential and just where the world's going in a very good way, very positive way.ADRIANA: Totally. That's awesome. Those are great words of wisdom. Well, thanks so much, Rob, for geeking out with me today, y'all. Don't forget to subscribe. And be sure to check the show notes for additional resources and to connect with us and our guests on social media. Until next time.ROB: Peace out, and geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:Luiz Aoqui (he/him) is a senior software engineer at HashiCorp working on Nomad.Find our guest on:X (Twitter)MastodonLinkedInFind us on:All of our social channels are on bento.me/geekingoutAdriana’s X (Twitter)Adriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:Go (Golang)HashiCorpHCLNomadJava JARAmazon EKSGoogle GKS (GKE)Microsoft AKSNomad Agents and ServersVault (HashiCorp)Terraform (HashiCorp)PodmanManaged KubernetesKubernets OperatorNomad Operator patternnomad-ops projectKoyebkreconcilerKubernetes CRDNomad 1.6Nomad node poolsKubernetes node taints and tolerationsKubernetes node poolNomad datacenterNomad EnterpriseNomad scheduling algorithms (spread, binpack)RFCNomad on Kubernetes Tutorial - Kelsey HightowerDocker-in-Docker (DinD)K0sBlog post: Running K0s on NomadetcdKubernetes controllersKubeletk0s on ARM machinesHashiqubeVagrant (HashiCorp)Consul (HashiCorp)Jaeger (tracing tool)k0s - cgroupnscgroupns flag - Docker runLuiz Aoqui on On-Call Me MaybeLightstepHoneycombZipkinNomad PreemptionDProfGo OTel auto-instrumentationeBPFTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And geeking out with me today is Luiz Aoqui. Welcome, Luiz.LUIZ: Hi, Adriana. Nice to be here. Thank you for having me.ADRIANA: Thanks for coming. Where are you calling in from today?LUIZ: I'm calling from Toronto as well.ADRIANA: Woohoo.LUIZ: Representing.ADRIANA: Fellow Brazilian in Toronto. Awesome.LUIZ: lots of us here.ADRIANA: Yeah, there are lots of us here. So, we're going to start off with some lightning round questions before we get into the meaty bits. So let's get ready. Be prepared. I swear it's painless. Okay, question number: one are you left-handed or right-handed?LUIZ: Right-handed.ADRIANA: All right, question number two: iPhone or Android?LUIZ: Android.ADRIANA: Number three Mac, Linux or Windows? What's your preference?LUIZ: Going to say Mac for now.ADRIANA: For now. Awesome. Favorite programming language?LUIZ: Go. Pretty big favorite of mine.ADRIANA: Awesome. Dev or Ops?LUIZ: I like both, but I have to say dev.ADRIANA: Cool. No, wrong answers. JSON or YAML?LUIZ: I think I'll pick YAML. Just more friendly.ADRIANA: Fair enough.LUIZ: I hate the lack of dangling commas on JSON.ADRIANA: Yeah, I agree. I hate that too. And then finally, do you prefer to consume content through video or text?LUIZ: Text. Yeah, I get very distracted watching videos.ADRIANA: Yes. Same. Awesome. You survived the lightning round. By the way, I should mention we had someone on here who when I asked her JSON or YAML, she said HCL.LUIZ: Yes. That's why I had it in my back of my mind, like, oh, I only have two options.ADRIANA: So I thought that was pretty funny when she mentioned that. And then I thought of you, because for those who don't know, Luiz works at HashiCorp, he is a Nomad developer, so HCL fits well into the type of work that you do. So I wanted to start off with you being a Nomad developer. Tell us a little bit like, for folks who aren't familiar with Nomad, tell us a little bit about what Nomad is.LUIZ: Sure, yeah. So Nomad is a workload orchestrator, and I know that doesn't mean a lot to many people. So the goal of an orchestrator is to basically get your assets...So, like, your developer team build things like Docker images, binaries, Java JAR files, whatever. They produce some kind of artifact out of source code. And then you have your infrastructure team that is responsible for running your infrastructure, building servers, configuring machines, all of that. And then in the far end, you have your users that are trying to access your product, trying to access your application. And then the orchestrator sort of sits between your development team and your infrastructure team to make sure that whatever artifacts gets produced, it's running on those machines. So it helps finding where to run things like figuring out what's the best server to run this application, or doing things like upgrading and deployments and all the sort of lifecycle management of your application. That's the job of the orchestrator. That's what an orchestrator do.ADRIANA: Awesome. Yeah, and I think that's such a great way of explaining what an orchestrator does, because for folks who are familiar with Kubernetes, I mean, Kubernetes is an orchestrator as well, specifically for containers, whereas Nomad gives you that breadth of, pretty much orchestrate anything, more or less. But it is very easy to kind of forget all of the gnarly things that happen behind the scenes in these orchestrators, like all the hard work that they're doing in order for them to operate seamlessly.Now, I've played around with both Kubernetes and Nomad, and I have to say one of the things starting on the Kubernetes side and then moving to Nomad, moving to Nomad was actually a lot easier because you kind of dealt with the complexity of Kubernetes moving down to Nomad. You're like, "Oh, it's like the simplified...everything's simpler." And it runs in a single binary. It can run in a single binary on your machine and you can get started easily, whereas Kubernetes is more of a beast. I mean, yes, you can have really complex setups with Nomad, of course, and that's probably how you have it in production. But as far as I think the barrier to entry when it comes to Nomad is very low, which I think can be very appealing.LUIZ: Yeah. Complexity is an interesting discussion because complexity sort of means different things for different people, I think, when thinking specifically on this Nomad versus Kubernetes discussion, I think there are a few things to consider when you think about complexity of adoption. Let's say when you start, if you starting from a managed service for Kubernetes, like EKS, GKS, AKS, that's like one click, and then you have a cluster. So like, oh, there's no complexity there. And that's how most people nowadays consume. Kubernetes is through a managed service. So that almost basically removes the barrier of entry for those that are able to use a managed service, of course.But then it gets the complexity of understanding how to use those systems. Like, okay, it gets provisioned for me. There's some cloud magic happening behind the scenes. I don't have to deal with that. But now you have to run the system. Now you need to think of how do you map your team's workflow to that new tool? So there's all these different concepts in Kubernetes that I think is part of the complexity. There's all these different tools that you can use. Having a broad ecosystem is great, but it can also lead to some confusion about when do I adopt to do this or when do I do that, how do you bring all together to eventually reach your end goal?Which is like, I want my users to be able to access my product right. When I think complexity, I think more on that sort of day-to-day operations, like understanding what's happening with the system. And I think that's when sort of Nomad becomes simpler just because it has a smaller surface area for people to interact. You write your job, you run your job, and there you go.But like stepping back a little, the complexity of Nomad comes in on the deployment part because we don't have a managed service of sorts. Like, okay, now you need to understand what are Nomad agents, what are Nomad servers. Now you need to manage their state, now you need to manage upgrades. And this can get complex in that sense. So I think that discussion of complexity, I find it very interesting just because of this duality of like, okay, what am I trying to do? Am I actually running the cluster? Am I actually just using it? And somebody is provisioning...nowadays we call the platform teams. Is there a platform team running a cluster for me? And so yes, it's no much simpler than Kubernetes, I guess, depending on what you like, depending on where you're kind of like depending on what complexity you're talking about.ADRIANA: Yeah, that's so true. That's a really good point. I just want to go back to a point that you made earlier about the fact that there's all these Kubernetes managed services, but there's Nomad managed service, which is kind of interesting, because if you look at various Hashi offerings, there are managed services for a bunch of stuff. So why is it that there is no managed Nomad right now, even offered through Hashi's own cloud?LUIZ: Yeah, it's something that definitely part of the plan, is something we think about. But there's quite a few challenges for Nomad, specifically compared to other HashiCorp tools, is that we have a big competition with Kubernetes. So if you have a managed Vault, that Vault, this is basically the only tool you have.You have a managed Terraform, and there are other tools for infrastructure-as-code, but like, Terraform is one of the big ones. For Nomad, this competition is much stronger, in that sense. Where you're...so, that's mostly my personal opinion.But thinking of a managed Nomad service versus a managed Kubernetes service, it gets into that point of how much value that actually becomes. Because part of the benefits of Nomad is the flexibility and the flexibility of running different environments, flexibility of running different workloads. But those workloads, you sort of need to have control over infrastructures. Like, I want to use Podman, so I need machines that have Podman installed.ADRIANA: Ahh, okay, got it.LUIZ: Going back to that discussion of managed versus self-hosted, it's a spectrum, right? Like if you go to managed, it's simpler to operate, but it's probably more expensive. And you also lose flexibility, whereas in self hosted it's harder to manage, but you get full flexibility and probably cheaper. So there's that aspect that sort of like, we need to figure out.There's also something to consider about costs and things like that. Because if you have a managed Kubernetes service on AWS, AWS is sort of taking the heat of like, they can sort of discount the compute because they run the compute.But now if you have an external managed Nomad, you need to account for the price of that service plus whatever infrastructure you're using. And so it kind of becomes like a pricier solution. And so figuring out the business part of that can also be a little challenging as well. So there's all of these little things to consider, but I have to see what happens in the future.ADRIANA: It's a really great explanation. Thanks for clarifying. Another thing that I wanted to ask you about is, I think one of the big things that Kubernetes has that you don't see, I don't think a direct translation for In Nomad is that you've got the whole Kubernetes operator concept, whereas as far as I know, there isn't that concept quite in Nomad. I've seen a few blog posts where people try to replicate it to a certain extent, but it's not quite the same thing. Can you comment a little bit about that? LUIZ: So that's a very sort of common question to see. And I think there's this idea that you need to think about that operators is just a pattern that you can implement in pretty much anything. So the idea of listening for events and responding to them and that's something you can do with Nomad. And actually a few projects that I've seen do that. I was googling for the name. There's a project called nomad-ops that implements the operator pattern in Nomad using some of the functionalities we build.There's also a company called Koyeb. They are a platform-as-a-service company that use the operator pattern. And they have a library called kreconciler, I think it's called, that helps you build this sort of operator paradigm functionalities. But it's also important to think that on Kubernetes, it's not just operators that are the main thing. When you talk about operators, you're always associated with a CRD because that's the data. So you have operators being the logic, CRDs being the data. And that sort of helps guide your end goal sort of based on those two concepts.So in Nomad, you can do the operator. We're building things that can help you do that sort of things. One of the challenges, like how do I access the Nomad API from my task? And we're building like, now you have like a socket that you can use to talk directly with API. We are building workload identity, so you don't have to worry about ACL tokens or anything like that. It's like we're building things that help people create the sort of operator paradise in Nomad.But the CRD sort of becomes a bit of a challenge because we don't have that concept of as extensible as Kubernetes has. But you can sort of have that Nomad variables and things like that. You can kind of get around those challenges. But I think there's no another point of CRDs is that they're sort of standardized, right? Let's say the Prometheus operator expects to have this or generates specific CRDs. And then a Grafana operator can sort of rely on those CRDs to do stuff automatically. So this type of standardization, we don't quite have that yet.ADRIANA: Right. Interesting. That's really cool. I'll definitely be sure to put those two projects that you mentioned in the show notes for folks to refer back to those. Another thing that I wanted to touch upon, because I believe you and I talked about the fact that there was a new version of Nomad that just came out. So what version just came out? And what are the exciting things that we can expect to see in Nomad?LUIZ: So Nomad 1.6 just came out and the main feature is called node pools. And something that I worked on, so apologies for any bugs. The idea of node pools is that it allows you to create...sort of like, segment your clients into groups, into pools.So a bit of Nomad background very quickly. So you have two types of machines in Nomad cluster. You have a servers. It's like your control plane. They do the scheduling, they store state, they sort of do all the global view things of your cluster. And then you have clients, they're like talking to the servers to get information about what that specific client needs to run.So the client is sort of the data plane is the component that actually run things, so actually runs your Docker containers, your JAR files, whatever. And so one of the challenges we've had in the past with Nomad is that it's very hard for you to sort of associate a group of, let's say I have a group of clients that I want to run my backend services.And you can do something like create a constraint that says, okay, my back-end service only runs on machines that have this specific metadata. But doing the opposite is kind of hard because constraints are like, you need to tell a job what the constraint is, but in order to sort of prevent others from accessing those same machines, you need to create like a negative rule. So you have to say, okay, this machine, this job runs on these machines, but every other job is forbidden from accessing those machines. So you sort of have that dual constraint type of thing. So it's very hard to manage that to get a consistent scheduling outcome because of like, if you forget a constraint rule, now your job is running somewhere that it wasn't supposed to be.So with node pools, you can put a new configuration on each client saying which pool it belongs to, and then on your job you'll say, okay, this job runs in this pool. Now only jobs in that pool will access those clients, and those jobs will only run in those clients. So you kind of create a sort of segmented part of your infrastructure that is reserved for specific jobs.ADRIANA: I want to say what you were describing without the node pool is kind of reminding me of Kubernetes node taints, where you can say what can run where.LUIZ: The idea is to have a very simple solution. So it's like this node is in this pool, this job runs in this pool, and that's all you have to do. So there's not a whole lot of configuration that you have to do. And in addition to having this sort of segmented view of infrastructure, node pools can also hold configuration. So they are a first-class concept. In Nomad, for example, one of the workarounds people used to do to get around this constraint problem is to use data centers.So in Nomad, the idea of a data center is just like a collection of nodes. And if you have an availability zone, that sort of becomes your data center. And people would kind of hack around the problems using data centers, so they have like a data center for apps, which doesn't quite match the intention behind it.But the problem is that data center is sort of just like a metadata, so you cannot have specific configurations per data center, things like that.ADRIANA: Okay.LUIZ: But with node pools, you can attach like you can put a description on your node pool, you can put metadata in the pool to sort of create more information about what this pool is used for. And then in Nomad Enterprise you can actually have different scheduler configuration per node pool. So for example, you can have a pool that uses the spread algorithm and other pool that uses bin packing so you can adjust how the scheduling is done per node pool. So there's a bit of extra customization that you can do per pool and that could be very helpful in several cases.ADRIANA: That's awesome. It's interesting because there's this recognition of, like, oh, people were kind of trying to hack together the concept of a node pool by using these data centers. So it sounds like there's this recognition of oh, users are trying to do this. Why don't we formalize it and turn it into a proper solution? What I like that you had said earlier also, which I think it feels like it is a general philosophy with Nomad is basically going for the overall simpler solution. Like, don't try to overcomplicate. Just go with the base thing. That works pretty well and we don't have to drive ourselves mad.LUIZ: Mmhmm, yeah, exactly. Yeah. And that was a big thing during the development phase because a bit of background on how we develop features at HashiCorp, we start with an RFC. So whenever we want to implement a feature, we write down the description of the feature and sort of send to the we first sent to our immediate team, then to the whole company for feedback. And during that process I had gigantic ideas like, oh, maybe node pools should be dynamic and then you can dynamically add and remove nodes from the pools. But that sort of adds so much complexity with questionable value.So that's like part of the feedback I got from the team was like, let's start simple, let's start solve the problem at hand and then we can expand if the need arise. But yeah, this idea of simplicity, trying to make things easy to use from day zero, it's very important to us.ADRIANA: Awesome. That's very cool. I want to switch gears now because there's, like, two things that I'm hoping that we'll have time to talk about still because there's so much cool stuff to talk about, but I want to switch gears quickly to a collaboration that you and I did, which was really fun. It came out of just me having a wild idea that came out of nowhere, where basically I thought, "Hey, wouldn't it be cool to try to run Kubernetes on Nomad?" Because there's Kelsey Hightower's well known GitHub repo where he's running Nomad on Kubernetes.So I posed the question, "What if you can run Kubernetes on Nomad?"And I thought, "Maybe let's not try to go too crazy here." And so my idea was, like, I want to find a lightweight Kubernetes distribution that we can run on Nomad, something that's hopefully already containerized, because trying to run Docke-in-Docker is kind of a nightmare if you try to do it yourself. And so there's this distribution of Kubernetes called k0s that comes in a Dockerized version, which seemed relatively straightforward to deploy on Nomad.And so I reached out to you when I came up with this idea, and then you helped me through a bunch of the troubleshooting, so I just wanted to talk to you, have you share your experience around this collaboration. Yeah, just thoughts.LUIZ: Yeah, it was pretty fun. Like a lot of learnings, I think, in terms of just understanding how things work, because k0s is pretty cool project in the sense of like, oh, you just run that image and you get a container with everything sort of there for you. But I found it a nice learning lesson of debugging of when things don't work. So normally you would expect things just "docker run" and it works. But what happens when it doesn't work and having to debug and going through logs and sort of combing through those log lines. I found it a bit challenging because having everything in an image, it's easier to start things. But then when you need to debug and you have your etcd logs at the same time as your controller logs, at the same time as the Kubelet logs, it all sort of juggles together and it was very hard for us to sort of comb through that and understand what's actually failing.ADRINANA: That's so true. That's so true.LUIZ: And then you have retries, right? So you see an error message and then it retries and then there's an error message again. But is it the thing that is retrying that is the problem or is it trying to call something else that failed before and then the log just sort of disappeared from the history? Just because that part, I found it very interesting.ADRINANA: Yeah, I totally agree. And it was funny because I did the classic rookie mistake of like, well, of course this thing works in Docker. Let me just try to deploy it in Nomad. And then I realized I was getting all these error messages where the Kubelet was not starting, which you kind of need that for Kubernetes. So it looked like it started up in the Nomad job, right? The Nomad job deployed successfully, but the actual thing inside the job was not running correctly.So I neglected to try to run the k0s locally on Docker and then discovered a bunch of stuff where initially we were running into all sorts of issues, too, because running on M1 Macs, everything is special. I love having an M1 Mac, but, my God, there's all these little annoying considerations. So that made it extra complicated. But then once we got it running standalone in Docker on the M1, then we were able to port it over.And it was interesting because for me, I always like to try everything in Hashiqube, which is like this full-fledged Nomad environment where you have, like, Nomad, Vault, Consul all running together. But it's provisioned using Vagrant. And on the M1 Mac...normally you'd provision using a VM in Vagrant...but on the M1 Mac, Vagrant does not play so well. So it's Docker as the provisioner. And so you're basically running Hashiqube. So you're running Nomad in Docker. And then you're deploying Kubernetes as a Docker image and then deploying our test app, which was Jaeger. Was Docker in Docker in Docker. It was like, I don't know, three or four layers of Docker.And then you took the more pragmatic approach of, like, let me just run this using the Docker binary. Sorry, the Nomad binary. Much easier.LUIZ: Yeah. That applies like several scenarios when a GitHub issue shows up, people describe their entire cluster and environment and give us a ginormous job file. And usually my first step is, okay, I need to reduce this, you need to boil down toADRIANA: It's so true.LUIZ: What's the actual problem? So, okay, let's try to remove, let's say, half of the job that I don't care about.ADRIANA: Yeah, I think that is a very sound approach.LUIZ: Let's just run a dev agent, see if that sort of reproduces the issue. Normally, my first step is to reduce as much as I can and then start adding things. So, yes. I don't know. Dev agent for me, Nomad is like my direct go-to anytime I need something. Nomad, "nomad agent -dev" and start from there. Yeah.ADRIANA: Reduce the noise as much as possible and then start building back up until you figure it out. This is the wall I actually hit. So, yes, lesson learned to you all. I should know better. I've been doing this long enough that I have found myself in situations where I want to do everything all at once. And then I'm like, strip, strip, strip, strip, strip all the things until you get to the actual problem. But this was a fun little collaboration.And then there was one component. What was it? The C groups namespace where there's a Docker configuration that Nomad did not support. And so this is where it helps knowing somebody that works on Nomad because Louise was able to make, like, a little fix to accommodate. It's not part of the Nomad product. So you will not find this as part of standard Nomad. This was just so that we could see if we could get this running with this configuration.LUIZ: That part is very...on the surface, it's like, oh, it's like configuration value that it passed to "docker run" on the Docker CLI, that's just a flag that is set. But what it actually does, it's much deeper onto the environment that you're the I forgot the exact flag.I think it's cgroupns. And then you need to put host.ADRIANA: Yeah, I think that's the right one.LUIZ: But the tricky data is that Docker and cgroups, they do, like, weird stuff that Nomad sort of needs to work around that to make some of the Nomad things work. So, for example, resource isolation. Nomad uses cgroups to sort of enforce that. No matter what task you're running, no matter if it's a Docker container, a binary, JAR file, we use the cgroups because that's the common layer. But the way that Docker does things, it kind of hides that from you. So you as a developer, sort of needs to work around all the things that docker does with cgroups to get that to work.And so even though it works, that configuration, it's kind of dangerous in the sense that it can lead people to break other stuff without realizing. And so that's like, yes, we should support this, but not this naïve approach that I did.ADRIANA: Yeah.LUIZ: Luckily, this is such a sort of common problem that we'll probably have a better cgroup handling in a future release. And once we got that, then we'll be able to support that feature. But for now, it's on my sad, unmerged PR.ADRIANA: But it's interesting because it was a good learning, right? Because it was like, hey, we got this to work with this special unmerged PR. But then it kind of led to more questions. Right. And I think this is a really great lesson for anyone. Whereas, yeah, this might seem like, oh, this is an easy solution, but what are the repercussions? And that's why pull requests exist...LUIZ: Exactly.ADRIANA: ...so that we can mitigate against weird things happening, because you just simply do not know what the side effects are going to be from, like, oh, I added this little flag. What's the worst that could happen, right? So, anyway, it was a really cool side project. I'll provide a link to the blog post where I detailed our adventures in the show notes. And then the final thing that I wanted to talk about, because when you were On-Call Me Maybe, it was one of the reasons why we brought you on was to talk about how you had played around at one point as part of a hack week to try to add OTel instrumentation to Nomad core. And this was, I guess, over a year ago. So I was wondering if you could talk about what you've learned a year on what the status of that is right now.LUIZ: Yeah, cool. Yeah, that was still pending. I would say it's a very side project of mine, just like an exploration. I think I try at least three times now, try to get some OpenTelemetry to Nomad. And every time I learn something new, which is great and sort of like, builds on top of the previous attempt. So, like, a bit of history is, like my first attempt was sort of a very big view of, like, I want to instrument whole Nomad. I want to be able to create this sort of trace and spans from, like, I submit a job, that job gets scheduled, that gets picked up by a client, and that client starts a test driver, and the test driver calls Docker to start.I wanted the whole flow as trace and spans and all of that. That turned out to be a terrible idea just because I want to say it's not doable, but it takes a lot of code changes to get to that point. So that's the first learning. Don't try to do everything at once. And then my second attempt was more focused around not exactly OpenTelemetry in Nomad per se, but like helping people using OpenTelemetry and running things in Nomad to get information. Now I forgot what the name of that component is, but it's like a way for if you have an application that uses the OpenTelemetry SDK to automatically pick up information from Nomad, like the allocation ID, the job name.So you use the OpenTelemetry SDK. There's like environment variables that are sort of standardized.ADRIANA: Yeah.LUIZ: So provide those things automatically.ADRIANA: Oh, yeah, because I think there's, like a similar thing in Kubernetes where you can automatically grab from your Kubernetes pods.LUIZ: Yeah, there's a whole spec for that. I forgot the name, how it's called, but it's a way to sort of automatically infer information from different sources based on either environment variables or API calls. So I kind of hack around that and sort of works. There's another set branch that I didn't emerge with this work, but the challenge there is sort of like it's kind of hard to tell what information is relevant because you also don't want to shove a bunch of things because it's going to increase your network packet size. It's going to generate a lot of extra information that you may not care about. So I'll have to build a way for you to customize which information you want. So that's where I put a pause on that.And then my last attempt after talking with you, Ted, and some other folks on the OpenTelemetry community, I learned that don't try to boil the ocean, don't try to instrument everything at once. Focus on your core business logic. Start there and then you're going to get a lot of value from that already and then you can start building on top of that. And so my latest approach was like, okay, I cannot create a whole trace. I cannot create that relationship between traces, but can I use metadata to connect them? Probably I should explain this, but one of the challenges with OpenTelemetry in Nomad is that OpenTelemetry, and more specifically the distributed tracing aspect, is sort of focused on microservices and network requests and sort of keeping a track of those network requests.But in Nomad, the complexity is sort of built in into the Nomad binary. So the complexity comes almost like from local function calls rather than network requests. And so if you try to create spans for function calls, you get like tiny traces of a few milliseconds that are not really useful and it just generates a huge overhead. But what helped me there was understanding this notion of like, oh, I don't actually need to connect the traces per se. If they have the same metadata, then sort of like whatever platform you're sending those traces to, like Lightstep, Honeycomb or Zipkin, Jaeger or whatever, then you can start querying traces that have the same metadata so you don't have an explicit connection between your traces.But the metadata becomes a way for you to start to understand what happened. And so that was the last attempt that I did and it was quite successful. It works very nice in terms of trying to understand the inner workers of the especially the Nomad scheduler, because that's sort of like the magical box. You run a job and suddenly you have a bunch of allocations for who knows what reason. And so my goal was trying to understand what happens in there. Because if you look at the source code for Nomad, people that know Go, who like to get an adventure, search for a function called compute group in Nomad's GitHub and try to understand the function.And then come explain to me once you understand, because that's the function that gets a job and generates the allocations. So it looks at the clients, looks at what allocations already exist. And it's sort of like the central point of all Nomad features, more or less. I think people don't realize how many features Nomad have, but things like preemption deployments, disconnected clients, all of this sort of needs to take it into account when you are scheduling things and it all comes into that function.So like, Compute Group is my nemesis, and every time I need to touch that, it's like I need a fresh cup of coffee to go there. But yeah, my goal is like, okay, can I make this function more understandable using telemetry? And it helps in quite a bit in some ways, but there are things that this process is just complex for this. You kind of need to embrace that sometimes.ADRIANA: It's interesting, right? Because you start projects like this, you're like, of course it's going to be easy to instrument.LUIZ: Yes, there's an SDKADRIANA: It's like the k0s on Nomad thing. Of course it's going to be easy. And it's like no.LUIZ: There's tutorials, and there's all these different materials. Just go install this SDK. But no, it's a very different use case, right? Normally you come from this microservice architectures and then you're trying to instrument the communication patterns between them. But I'm trying to do something very specific that I don't think would apply to most people using OpenTelemetry. So, yeah, it's almost like...ADRIANA: And you said a lot of the processes are asynchronous, too, which makes it kind of work with, right?LUIZ: Yeah, so like when you have...the lifecycle of the job, right? Like a new Nomad job run that generates a HTTP request to whatever client or server you're talking to. That request needs to go to the leader, so there's another request going to the leader. But then once it gets to the leader, then there's a bunch of asynchronous stuff happening. So it creates an evaluation. That evaluation gets picked up by what we call it a worker, like a scheduler worker that does all the computing. Once it figure out which allocation needs to get run, then a client picks up. There's no direct network request that covers the whole thing. It's a bunch of put in a queue somewhere, put in a broker somewhere, and then that gets picked up. So that's sort of when you lose your trace a little bit.But the tricky thing about Nomad is the network request is like the easy part. The complexity is like, what happens after you receive that request. So that was the thing that I wanted to instrument, was like, yeah, network requests, yeah, they happen, it's fine. I know who is talking to who. But inside each process, that's sort of like where the challenge lies. Like, okay, how do I get visibility and what's happening right now in there? And that's like, I don't know, it sounds like a fourth pillar, perhaps. We have metrics, logs, and traces. Maybe there's something new that should exist there. But yeah, that's challenged it's like understanding what's happening inside the process.There are a few tools for debugging, like DProf and things like that, but they're very low-level in a sense and you don't always want to run those sort of additional instrumentation in production. That was the challenging part.ADRIANA: Yeah, it I did see something that came out this week where there's, like I want to say there's, like, some sort of go auto instrumentation air quotes, maybe not air quotes with eBPF that can help give some additional insights where that could be a game-changer.LUIZ: Yeah, that would be pretty cool because eBPF, you can sort of hook into anything like any sort of system call or whatever that combs from your program. Yeah, that could be interesting. Specifically like, thinking of the Nomad case when the scheduler is very complex. But there's also a lot of complexity in the client because, oh, I need to run a Docker container. Cool.But it's not just that. Especially in Nomad, we have templates, artifacts, volumes. So you need to mount a volume, download a file, you need to render a template, you need to fetch tokens from console involved. So running a simple container, there's like a whole lot that needs to happen beforehand. And we call those like lifecycle hooks. So you can have things that happen before the task starts, things that happen after the task starts. And a lot of those interact with the operating system. So being able to instrument sort of like what's the Nomad agent trying to do against the OS could be very nice.ADRIANA: Yeah, cool. I think there's definitely more work to be done in that area. But I'm glad that you've continued experimenting. Even if it's not gone, maybe as far as you would like, I think it's still progress, so, you know...LUIZ: Yeah, it's all learning.ADRIANA: ...it's awesome. That's awesome.LUIZ: Like, I think it helps something different, I guess. Something different to learn something different. It's always good to keep up to date what's happening. And a lot of people are starting to adopt OpenTelemetry more. So even if it never comes, that OpenTelemetry is integrated into Nomad core...But I think it's helpful to at least understand because my target audience will maybe use OpenTelemetry on their stuff. And whenever I talk to them, I sort of need to understand what they are doing and how things work. I know if somebody comes and open a niche and say, oh, I'm trying to run the OpenTelemetry Collector in Nomad, I would need to know what they mean. And having this sort of exploration is very helpful.ADRIANA: Absolutely. Cool. Well, we have come up on time. We could keep talking about this forever, honestly, so we'll have to have you back again. Thank you so much, Luiz, for joining today for geeking out with me. Y'all don't forget to subscribe, and be sure to check out the show notes for additional resources and connect with us and our guests on social media.Until next time...LUIZ: Peace out and geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:Renata Rocha has been working with technology since 1997, first as a sysadmin. She then found her passion for Cloud Engineering and never looked back. She has been at Slalom since 2019 and loves everything about the Cloud, Platform Engineering, and the endless possibilities it brings us.Find our guest on:MastodonLinkedInFind us on:All of our social channels are on bento.me/geekingoutAdriana’s X (Twitter)Adriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:International Lefty DayNoYAML.comHashiCorp Configuration Language (HCL)Python (programming language)HashiCorp (Hashi)HashiCorp TerraformHashiCorp NomadSysadmin Appreciation DayMainframeRoyal Bank of Canada (RBC)COBOLJob Control Language (JCL)Platform EngineeringServerlessData centerCyberpunkShopifyBellCentral Office (Telecom)Jacarepaguá (Rio de Janeiro)Rotary dialHow to use a rotary dial phoneDigital transformationADHD hyper-focusPomodoro TechniquePomodoro TimerTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, Reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And today geeking out with me, I have Renata Rocha. Welcome, Renata!RENATA: Hi, nice to meet you. I'm Renata Rocha. I'm a principal of Cloud DevOps security, actually. Platform engineering with Slalom Build. I have been at Slalom since 2019. I love my job, I love what I do, and I love technology.ADRIANA: Awesome. And where are you calling in from today?RENATA: I am based off Toronto, Canada. Like Adriana. Okay. Yeah.ADRIANA: Awesome. Yay fellow Torontonian, fellow Canadian and fellow Brazilian.RENATA: Fellow Canadians and fellow Brazilian as well. Yes.ADRIANA: All the things. All the things. Cool. Okay, so we're going to start up with some warm-up questions. First question, lefty or righty?RENATA: I'm a lefty. I'm a very proud lefty. I do everything with my left hand. My right hand is absolutely useless.ADRIANA: You're the first lefty that we've had, and I'm so happy to have a fellow lefty on the show because I, too, am a proud lefty. And International Lefty Day is on August 13.RENATA: I did not know that. That makes total sense. By the way, awesome.ADRIANA: Super excited for Lefty Day. I always forget about it until after the fact, so I'm hoping I'll observe it this year. So lefties unite.RENATA: Lefties unite.ADRIANA: Okay, next question. iPhone or Android?RENATA: iPhone. It's interesting because I actually would prefer an Android, but it's been a while since they released a very small flagship Android phone. All the flagship Androids are huge, bulky phones and I have very tiny hands. Okay. So I have been an iPhone user because it's the smallest flagship phone I could find in the market. I would go back to an Android any day if they release a smallest Android flagship phone. Okay. Just because of that.ADRIANA: Yeah, fair enough. I actually miss the really small phones. Do you remember those little tiny Nokiavphones that everyone used to have?RENATA: Absolutely. I love those. Oh, yeah. They were so cute, right? Yeah. I'm under five foot tall. Okay. I'm very petite, so my tiny hands would rather have a small phone that fit into just one hand. I don't have, like, two hands to type. Yeah. They don't think about people like me when they design phones.ADRIANA: Right? I do feel like we're outnumbered by taller people here in Canada. Awesome. Next question. Mac, linux or Windows?RENATA: Oh, great question. I love Linux, but I have been a Mac user for a number of years. I think Mac offers a great desktop environment with also a great Unix like system. So I can do everything in terms of programming development on a Mac with a pretty environment in front of me. But for systems, my setup, my servers, they are all Linux, obviously. Windows is a no, no, I don't touch that. Okay.ADRIANA: I'm kind of the same with you there, actually. I cry every time I have to touch a Windows machine. I'm sorry, Windows people, don't mean to offend, but it's just not my thing.RENATA: Oh, my God, I feel no, no, I feel dirty. Okay. Yeah. Awful. Ewwww...ADRIANA: Next question. Favorite programming language?RENATA: Python. Absolutely.ADRIANA: Me too. I love Python.RENATA: Python is beautiful. It totally makes sense. I spent many years thinking, like, oh, my God, I hate programming. I hate programming. And then one day I discovered Python and it was like, okay. I love programming.ADRIANA: Yeah, I agree. I feel it's like a pleasure to program in Python. It always makes me happy whenever I touch Python code, so I totally get you.RENATA: Yeah.ADRIANA: All right, next one. Dev or Ops?RENATA: Oh, Ops. Yeah.ADRIANA: I kind of figured.RENATA: I come from a sysadmin background. Okay. I feel like inside of me there is a sysadmin. I will always be a sysadmin Okay. So Ops any day.ADRIANA: And that will tie right nicely into our topic of discussion in a little bit. Okay, two more questions. I think I know the answer to this one. JSON or YAML?RENATA: YAML Yeah. JSON is weird, and I think you can do pretty much everything with YAML. YAML is just easier, but I don't know, that's tough. Yeah, you kind of have to do it. That's my favorite thing. Can I just say HCL?ADRIANA: Oh, yeah, I'm right there with you, actually, because I always tell people I feel like HCL is the love child of JSON and YAML, because I think it's got the nice organizational stuff of JSON without the clutter, which is what I like about YAML. It's uncluttered. So, yes, I am totally there with you.RENATA: It just works, in my mind. The first time I saw HCL was like, yeah, this is right. This just rings a bell. And yeah, I love it. I think it's beautiful.ADRIANA: Totally, totally agree. And for those who are not familiar with HCL, I think it stands for HashiCorp configuration...RENATA: HashiCorp configuration language. I think that's what it stands for. Yeah.ADRIANA: Yes, that's right. Which is used for all the Hashi products, which is super awesome. All right, final question.RENATA: Sorry...ADRIANA: Oh, no, go ahead, go ahead, finish the train of thought.RENATA: I was just going to say that I'm a HashiCorp fan girl. Okay.ADRIANA: You know what? I have become too, mostly. I know, like, you work with a lot of Terraform. For me, my HashiCorp fangirldom comes from working on Nomad, so I can totally relate.RENATA: Yeah, Nomad, very underrated product.ADRIANA: Very underrated. Totally agree. And maybe we can talk about that a little bit later as well. Final question. Do you prefer to consume content through video or through a blog?RENATA: Yeah, a blog. Absolutely. I like reading stuff. Okay. The video things...sometimes I'm on the bed and it's late at night and I don't want sound, I don't want to see things. And on a blog, I can just read at my own pace. Yeah, I prefer reading stuff.ADRIANA: Yeah, I'm the same way. Of all the people I've asked so far, everyone is like, blog over video, so I've yet to encounter a video person. I know they're out there but yeah, I've yet to encounter one.RENATA: Yet you're making a video.ADRIANA: But this is also going to be out in audio. So I guess then there's the question: audio versus video. I'm more of an audio person. Because then I can...I'm a podcast person, so I like to walk and listen to podcasts.RENATA: Um, I like video. I need the visual cues. I don't...podcasts feel weird for me. I need to see people talking.ADRIANA: Yeah, I can appreciate that. What I do like is video with subtitles.RENATA: Oh, that is perfect. Video with subtitles. Yeah, that is...see, because you can read and then you can also see people's faces. Yeah.ADRIANA: Yeah, I'm totally there with you. Cool. Well, let's move on to our main topic because, as you pointed out earlier, today is International Sysadmin Day. From your background as a sysadmin...RENATA: Today is July 28th. It's the last Friday of July. It has been International sysadmin Appreciation Day since the year 2000. Like I said, I have been a sysadmin in the past. Then I moved to Cloud engineering, platform engineering...how we are doing it today. And I feel like these practices, they wouldn't exist if it wasn't for the sysadmins of the early days. We wouldn't have these things if it wasn't for systems administration. So Happy Sysadmin Day. Although we won't be seeing this video until later, but yeah, Sysadmin Day.ADRIANA: Because you said Ops over Dev...What is it about sysadmin that you love so much that you think people really underappreciate?RENATA: The thing that people don't see is when you are a very good sysadmin and you know what you're doing, your work is supposed to not be seen. Okay. The sysadmin is supposed to be invisible because the system just works. And the sysadmin is someone that is not there because you don't need to see them. You only notice your sysadmin when things crash, when things are wrong, when things are broken. So if you are good at your job, you're not appreciated, you are not seen, you are not present. Okay? So that is the interesting part of being in a sysadmin. But it's very satisfying to look at your uptime and see that your systems are up, that things are working, that things are just fine. Okay. I love seeing all my monitoring statistics and seeing yeah, everything is so finely tuned, everything is working so fine.That ties a little bit into observability as well. Observability. I know you are an Observability person. Okay. So, yeah, like I said, everything I kind of branched off and became like a specialized feature of all the things that we used to do in systems. And today, of course, how computers evolved and how we do things these days. They became their own specific practices and fields in their own but I still love...I have this passion for the way it's a type of nostalgia in a certain way, of how we did things in the past. And this appreciation for people who are still managing systems, especially more old school systems are still in operation.I know there are people who still manage mainframes these days and that's such hard work. Okay. It's amazing. Oh my God.ADRIANA: It's a very specialized skill.RENATA: Yes. And you don't find many of them. Okay. Yeah.ADRIANA: Yeah. As a student...I don't talk about this much because it was very traumatic...but I had a summer job working in the mainframe department of RBC.RENATA: My God.ADRIANA: It was something...it was something...I will say in air quotes, a COBOL code change that it was like, I don't know, I think it was like commenting something out or whatever. And it took forever to push that through because you had the Job Control Language at the time. So the language to compile the language. And I was like, oh man. No, not for me.RENATA: As someone who has also touched some COBOL calls back in the day...Yeah, I understand what you're saying. Yeah, I have some feelings. Like I said, nostalgia. Not necessarily want to go back there, but I'm going to say that I have played with some virtual instances running systems just for fun. Okay, yeah,ADRIANA: Very cool. Very cool.RENATA: Very cool. Yeah.ADRIANA: So, as sysadmin, what do you think has been like the biggest change that you've seen over the years as technology has evolved? As your career has evolved?RENATA: Sometimes I talk to people that I interview that are still working in systems and they are moving towards platform engineering. And what I think of what I used to do, and the biggest paradigm shift is that you used to log in directly into the machine and debug whatever was going wrong and fix it. So you had this control over each machine that you were running. You were very close to the system. To be quite honest with you. That feels very good. It gives you a sense of control.And with Cloud and platform engineering, you don't do that anymore. You simply don't log into the machine anymore. And you have so many layers of abstraction, especially if you're running something like serverless code, something is broken, something is wrong. You just fix that on your Terraform code, for instance, and then you redeploy it. So you don't even go to the machine anymore. You just fix your code and you redeploy it and then it's back up online. So it is a different way of thinking on how you were going to troubleshoot the problems and how you were going to deal with whatever is running your code. So it works differently.It's a lot of abstraction on top of it and you are not so close to the server as you used to be and you are probably going to miss it when you start working with so many abstraction in front of it. Okay, I understand that, and like I said, I miss it. It was fun. But at the same time, it gives you a lot of power because you had to go machine by machine to fix the problems and now you can fix the problem in hundreds of machines at the same time. And that is so amazing.ADRIANA: It's so true. Yeah. Do you think that having been an old school sysadmin where you were physically touching your machines...do you think that gives you a better appreciation then for the types of things that are happening in the Cloud versus the newer folks coming into it who have not known sysadmin in that same way, where you're, like, sometimes going into a data center and physically touching a machine, do you think that gives you an advantage as a result?RENATA: I hate saying better or advantage, but it gives me a different perspective. Okay. I work with a lot of people who have never been inside a data center in their lives. And when I tell them that, yeah, I used to work inside a data center and it's freezing cold, it's weird, there are like these noises and it's kind of soothing, the noise of a data center. Okay. You just have to wear a jacket because it's so cold and they're like, "Oh my God, did you work at a data center?" Like it was the craziest thing they've ever heard and yeah, it was a great job, by the way. I liked it very much.Yeah, I would ride my bike to the office, if the data center was actually downtown. I would ride my bike there and then work there all day. We didn't have Windows because it was a data center. Felt like this box, like, surrounded by computers for hours. And then I would ride my bike back home. Yeah, great job. Loved it all. Lots of cables. It felt so Sci-Fi, so cyberpunk.ADRIANA: If anyone ever has had an opportunity to work at a data center or tour a data center, it is surreal.RENATA: It is surreal. Okay. It feels like living in the future. Okay. So disconnected from life. Like all the people walking around living their lives, and then suddenly you are in that box of computers. Right? Yeah. It's amazing.ADRIANA: And you're holding the keys to the kingdom...to the computer kingdom, because you've got the admin passwords, you could do some serious damageRENATA: Yeah.ADRIANA: As an Admin, even as a Cloud engineer. Same sort of thing. But I think there's like a different...it's a different feeling.RENATA: It is a different feeling. You have this power that like yeah, if you do something wrong, a lot of things go wrong, but at the same time, you feel so isolated from everything else that is around you because it's this black box, so completely separated from the life outside.Like, I have the story that I was at the Shopify data center and there was a city-wide blackout that lasted maybe a couple of hours. And I didn't know it was happening because the data center has backup generators directly connected to whatever. And until someone told me, "Did you know there's a blackout happening outside city-wide?"I was like, "No, I didn't realize that." Yeah. The entire city has lost power. Okay. Amazing. That's why I like working there.ADRIANA: It's so true. I have a similar story because when Toronto had that big blackout in 2003, I think it might have been before you moved here, I believe, right?RENATA: Yeah. It was before I moved here. Yeah.ADRIANA: So in 2003, we had that massive blackout that lasted like a day and a bit. And I was working at a client site. The client at the time was Bell, and we were in the building next door to their Central Office. So the central office where they keep all their network equipment and stuff, like phone stuff, right? And so because we worked next door to the Central Office, we had the backup power as well, because the buildings were attached. So when there was the blackout, the lights blipped very quickly, and then it was all good. And so we're, like, keep on working. And then at the end of the work day, we start getting messages, like, there's no power in Toronto. We're like, "What?" Meanwhile, there's, like, power in the building. The elevators were still working. Everything was fine. And then you go outside, it's like complete chaos. Everything's out. Traffic lights, subways. Yeah, so yeah, the nice little bubble.RENATA: The nice little bubble. Okay. And that feels very comforting. And like you said, it feels like a lot of power that you are in this place that has everything. It has like the fastest Internet access you can imagine. It has power even when the entire city is in a blackout. Yeah. It is fascinating. It is such a great experience. And although I don't feel I have an advantage or I feel better than the people who didn't have this experience, it gave me a different perspective. I feel like I have learned what I do today at a slower pace. And I saw the Internet growing since the very early days of the Internet, and I was inside. What was the Internet back in the day?ADRIANA: Yeah. So true. Such a unique advantage that's something like, people from our generation, I feel like it's something that folks like, my daughter was born in 2008. She's grown up in a world with computers. It's ubiquitous. Smartphones. This is stuff where we're growing up along with it. It's like, oh, internet. Oh, cell phones. Oh, smartphones. We're all connected. Whoa, this is weird. Like, social media was not a thing when we were growing up. I mean, good luck trying to reach somebody. They're not home, you can't reach them. And you had to know people's phone numbers.RENATA: Phone numbers? Yes. When I was a kid, we didn't even have phones. Landlines so easily this way. And a lot of people didn't have landlines. My husband only had a landline when he was like ten years old. People had to write letters to each other. Imagine that.ADRIANA: I had a pen pal when I was growing up, and I totally remember my mom used to tell me stories of when we were living in Rio....I was too young to remember...but she was saying there was for a time we didn't have a phone yet, so she had to use the neighbor's phone to make phone calls. It was not the easiest thing to procure. It was, like, kind of a process to procure a phone line. And these are things we take for granted now.RENATA: Yeah. Where my husband used to live, which was a neighborhood in Rio, Jacarepaguá. They had this phone in the center of the neighborhood where people used to go and make a call because people didn't have landlines where they lived. Yeah. Crazy. And you think of it like, oh, must have been like a very rural area. No, it was just like a normal neighborhood, actually. Kind of an upscale neighborhood. It was just because landlines were not everywhere back then. Okay. Not a lot of people had that.ADRIANA: And I remember rotary dial phones when I was growing up in Rio, too, which, when I came here, i's like, "What? It's not rotary dial phones?" I still have fond memories of those.RENATA: Can you hear the noise? (of rotary dial phones) I saw something fascinating recently. I went to a doctor's office here in Toronto, okay? And there was, like, this emergency phone, in the doctor's office, and it was a rotary phone, and there was a sign next to it teaching people how to use a rotary phone.ADRIANA: So cute. The things we take for granted. We're like, "Of course! That's how you use a rotary phone. And people are like, "Uhhuuhh..."RENATA: Yeah. Wait for the dialtone. After the dialtone, you put a finger on the number and to wait for it to come back. Good times.ADRIANA: Good times. I kind of miss rotary dial problems. Yeah. Fun memories. Fun memories.RENATA: They were cute. They were cute. Yeah. Yeah. We had a red one, like classic red rotary phone one.ADRIANA: Oh, that's adorable. It's better than my beige one.RENATA: Amazing. It was adorable. There was also the beige ones. They were also very classic. Yeah.ADRIANA: Yes.RENATA: But yeah, it's funny to see how things evolved, and they evolved very fast. And I think the beautiful thing about working in technology is how you see things going from, like, the old rotary phones and then the dial tone phones and then the Internet, which leads us to the different fields, the specializations we have on platform engineering, SRE, etc. And now the new field of AI that people are like, oh, my God, AI, etc.It's just a new field, okay. And we need to learn how to deal with this new tool, this new technology, and how it ties together into the other things that we do. And that's the beauty of it. Right? This is why I do this. This is why I'm so fascinated by technology. I love learning things. I love being immersed in it, and I'm absolutely fascinated about learning whatever new thing, whatever new tool that people bring us today, tomorrow, next year.ADRIANA: Yeah, I totally agree. It's interesting because I'm the same way. I'm like, "New technology. Giddy up. Give me more!" I want to learn more about it. And yet some people are so terrified of technology, and we see this across our industry. We see it especially, like, in large corporations, whenever there's big transformations, whether it's a digital transformation or DevOps, Agile, whatever, even bringing in Observability...there is such resistance to change that can get...it legitimately freaks people out, right? Because to a certain extent, I think we're creatures of comfort, and we like what we know and understand, and having to deal with something that's a brand new learning curve can be terrifying.But what kind of advice do you have for people who find themselves in this scary place where they're resisting change.RENATA: Try to find yourself in a place which is 50% comfort and 50% challenge. Okay? That's a very good measure of what will bring yourself still that cozy space where you feel, yeah, this is good. This gives me this warm heart, this warm embrace of things that I'm comfortable with, but also doesn't make you bored, because once you get bored, you don't have anything to look forward to. And this is what makes us human, okay? If we didn't have any challenges, if we didn't have anything to look forward, any goals in life, we would still be living in the Middle Ages. We would still be cooking...eating raw food. We wouldn't be working with computers today. And that's what makes us different from other animals, is what makes us special. Okay? Yeah. This is pretty much what I try to tell people that find that sweet balance of comfort and challenge, because also, if you do challenges all the time, you're going to burn out. You're going to exhaust yourself. It's unsustainable. So find exactly how much comfort you can at the same time while finding that amount of challenge that makes you want to do something new.ADRIANA: Yeah, I totally agree. That's such a great approach because that way, as you said, you're not, like, going all in on it, and then that can be super overwhelming. But getting to a point where it's just like, oh, okay, this is interesting enough. I'm going to poke into it and kind of build on it. Iteratively, I think, is a really great way to approach it.And I think in technology, I feel like because, the nature of what we do, it evolves so rapidly, it can be very overwhelming. So I think people get freaked out. Oh, my God, there's another thing that's changing, another thing I have to learn. So I think to a certain extent, you kind of have to just pick a thing that's interesting, that like, okay, I'll dig into this a little bit more, and then dig into that and get into this habit of always learning a bit. And I think what people freak out with is they think they have to learn all the things.I resigned to myself, to the fact that I don't know everything. I'll never know everything. But I surround myself with people who know things, and that's okay.RENATA: You don't have to know everything. It's impossible to know everything. And you will never be an expert on anything because things evolve very fast. And like I said, before, technology is very dynamic. What you know today is never going to be what is the new technology, new big new thing of tomorrow. So what you have to do is to pick something that you want. Okay, go choose. I want to learn Terraform. Something that I don't know.Okay, I know Terraform, but just an example. I want to learn TerraForm. It is something that I'm interested. Just pick and choose and learn a little bit. Don't throw yourself in like, "I'm going to learn everything about Terraform today." I tend to do that.So that's some advice for myself. Yeah, I'm totally like that.So learn a little bit of it today and then give myself some time. Tomorrow I'm going to do something that I'm comfortable with and then learn a little bit extra. Just iterative process of learning so I don't exhaust myself. And also it gives your brain some time to absorb the knowledge. You don't feel exhausted, you don't feel burned out. The burnout is real. I always tell especially young people because they feel they are indestructible and they can learn everything everywhere, all the time.And then sometimes the burnout kicks in and they feel, "Oh my God, I cannot learn anything anymore and I hate this. I will move to a cabin in the woods and never see people ever again."This happened to me and that's why I keep telling people that it will happen if you don't take care of yourself. Please take care of yourself. It will happen. So give yourself some time. Go slowly, slow, easy, and yeah, just pick a subject here and there. You don't have to learn everything.And also no one expects you to know everything because we know it's impossible to know everything.ADRIANA: Yeah, I totally agree. And I want to go back to your comment earlier about burnout, because I agree with you. When you're young...first of all, when you're young, you feel like you're indestructible, invincible. But also, employers take advantage of that, too. They're like, "Well, you don't have a family. You don't have a social life. You just finished university. Of course you can go and put in the long hours."And I think it's so important for younger folks to give themselves permission to not do all the things, to take it easy, to not have to hustle so hard that they burn out. Because the same thing happened to me when I finished school.My first role out of school was so intense that I burnt out really fast and I hated my life. But the only good thing that came out of it was I learned to say no, and I learned to defend myself and stick up for myself in terms of, like, I've had it. Burnout. My brain doesn't work. Need to take a step back. And I think more young people need to do that. Just because you are young, don't have a family, whatever, don't have a partner, does not mean that you don't deserve to have a life.RENATA: Yeah. My suggestion is always to find an offline hobby.ADRIANA: Yes.RENATA: Something that is completely unrelated to your work. Learn to run, to climb, to do Yoga, to paint. Go learn to play the drums and musical instruments. Something that is completely unrelated to your work. You don't need to do some unpaid work. That's what I'm trying to say.ADRIANA: I totally agree. What's your go to activity?RENATA: I like to run, I like to cycle. I do Yoga, I have a garden in the summer. I have a ton of activities that I do that are completely unrelated to work. And they help me not think about work. They make me healthy, they make me happy. They make me a person that is not only the Renata who is at work, the Renata platform engineer. I am a more complete person and I'm also happier. I have a life that stops at 5:00 pm.ADRIANA: That's such great advice. I totally agree. Because of my ADHD, I get the hyper focus, and so I have a really hard time peeling myself away from things, and I tend to obsess over unsolved problems and can't shut it off. And sometimes I have to take myself aside and say, "I give you permission to not think about work, to disconnect." And it's okay to give yourself permission to shut off and pursue your things. Like, life is not all about work. It's going to be there tomorrow and the day after that and the day after that. I think employers are just looking to make sure that you complete your deliverables, that you're reliable. And I feel like if you do those things, then you're in a good position.RENATA: It...I also find that once I started taking care of my mental and physical health after work and not dedicating myself to working 15 hours a day, which I was doing, at some point I became much more productive at work.ADRIANA: Yes.RENATA: Which kind of makes sense if you think about it. But when you're younger, you don't think about it because I'm healthier. My mind is healthier. My body is healthier. So when I start my day in the morning, I can kick right in, start working, do a lot of things. I can concentrate much better. I haven't been sick in years, and I used to get sick a lot more. I just work better if I do this. My hours in a day are in a better person that can do just work for those hours a day. And I was so drained out by the burnout that I was not giving myself fully to work during those during those 15 hours that I was working before. So, there's that.ADRIANA: Yeah. I find also, like, the act of time boxing your day. You're like, holy shit, I got to get all this stuff done by 05:00. So you're like, okay, I'm going to be super efficient, right? Because you want to get as much done as possible, and then it's like and plus you have something to look forward to. At the end of the day, it's like, hey, there's like, relaxation on the horizon. It's awesome.RENATA: Yes, absolutely. When I have to deliver something and I have a deadline, I like to use a technique called Pomodoro. Some people are very familiar with it. You do 25 minutes of work, and then you stop for five, and then you do another 25. And after some four of 25, you give yourself a longer break that allows your brain to process the work that you did some rest. And it makes me feel way more productive because I just focus and just works very well for some type of people. It works for me. And if you're listening, and you are not familiar with the Pomodoro technique, there are some timers you can use on your browser or on your code editor. Give it a try. It might work for you as well.ADRIANA: That is very cool, I hadn't heard of that. But I definitely...like, taking the breaks is so important. And again, ADHD brain is like, you will not get up from this until you solve this problem. But whenever I do force myself to walk away and take a break and I come back, I'm like, oh, shit, I should have done this before.RENATA: Try the Pomodoro. Okay. Because they force you to take a break every 25 minutes, and it's a 5 minutes break. And then when you come back, you're like, "Oh, okay, I can pick this up again." Okay. It's not a long break. It's 5 minutes. Okay? Usually I just do maybe like, some Yoga poses for five minutes, and then I go back. I love Yoga. It's a great thing for my mind. It makes me relax. And then I come back, and then I do it again, and then I keep doing that, and I don't know, I just write code beautifully when I do this.ADRIANA: Yeah, I'm going to check that out because that sounds like something I could use. And also, I'm a huge fan of Yoga. I don't think I'm nearly as advanced in Yoga as you are. My flexibility is crap, but I do enjoy it. It's nice to it challenges your brain because you're so busy trying to hold the poses, you can't think about anything else. So I think it's a lovely way to just unwind.RENATA: Yoga, as with anything...technology...is not about the flexibility. It's about the inner journey, okay? It's about learning, understanding your body and where you are today. It doesn't matter where you were or just where you were going. Think about yourself, your inner body, where your balance is. So don't think about anything else. It's not about flexibility. It's just about the journey. So if you enjoy it, that's what matters.ADRIANA: Yeah, that's actually a really good point, because when I first started Yoga, I see it really mad that I was like, I look like shit doing these poses. Right? And then I'm like it's okay.RENATA: Yeah, exactly. No one cares.ADRIANA: As soon as I got over that...nobody cares, especially when I do it at home, nobody's watching. So it's great. And you start to see some progress. I mean, you're competing against yourself, which I think is probably the most important thing. Are you improving? Are you getting something out of it? Are you enjoying it?RENATA: That's why good Yoga studios won't have a mirror, because you're not supposed to look at yourself. It's just supposed to feel yourself.ADRIANA: Oh, that's cool. Good to know.RENATA: I like it very much.ADRIANA: Yeah, I'm a huge fan. I try to do Yoga, like, once a week, and I feel it too. If I don't do it, the joints feel a little stiffer. I'm like, oh, I think I needed this.RENATA: Focus on the breathing techniques. They really help me when I'm feeling stressed out. When I am obsessing over a problem, I just try to focus, recenter, breathe. And that sometimes helps me solve some piece of code that I cannot point or like some architecture that I'm struggling to design. I have a huge problem using diagramming tools. Sometimes I have the idea on my mind, but I don't know where to position things correctly. And then I stop. Take a deep breath. Okay. Do some breathing exercise. And when I look at the diagram again, I'm like, oh, yeah, here, write this.ADRIANA: Yeah. The power of stepping away, taking a mental break from your work, cannot be underscored. It cannot be underestimated. Awesome. Switching gears a little bit. Well, I guess going back to something that you talked about earlier, the idea of embracing the new technologies that come our way and of course, the new and cool technology that is taking the world by storm now that everyone's talking about and either excited about or feeling threatened by is AI. So what is your take on AI?RENATA: I have a hot take about AI, which is it's not going to take away any jobs. It's going to be exactly like Cloud was a few years ago. It's going to create a lot of new jobs. AI doesn't create itself. It's not actually artificial intelligence. That's just like a cool name for it. There are lots of people working to generate those libraries. You've deployed the code, so it actually requires a lot of people, qualified people, engineers develop that. So it is a whole new field that's open for you. It's fascinating. It's very early days, so yeah, it's going to create a lot of new jobs. So if we embrace it with open arms and open mind, it's exactly like when Cloud was born that people were scared, oh my God, it's going to take away jobs and look at where we are today. So embrace it, learn it. It's great. It's going to be good for us. Just don't be afraid because it's a new technology. It's just a tool. It's not something bad or crazy. Yeah, that's my hot take about AI. It's just a tool.ADRIANA: Yeah, I totally agree with you. I totally agree with you. And I think, like, with any new tools, it can be abused or it can be used to really enhance your job. And I think folks who end up using AI tocheat at their jobs or cheat at work, right? Like using AI to write an essay, you're doing yourself a disservice because then you're not learning. I mean, you lose out in the end. Fine, you get the marks, but you still can't write. Versus using AI as an aid.Like, the example that I like to give is, like, you're writing something, you've written something out, but there's like a character limit. Feed your text into AI to like, hey, can you rewrite this so that it fits within the character limit? I feel like that's a perfectly valid use case for AI, because you wrote it, the concepts are there. It came from your brain. But AI has just taken that little extra burden off of you so that you can complete that last step where you can use AI as, like, inspiration, as a starting point for code. If you don't know a particular language, but you know how to code, but you don't know the nuances of that language. So AI can give you that starting point, but you still have to complete it.RENATA: You can use it as, like, a skeleton generator. Okay. But you still have to refine the results, and you have to analyze it to make sure that the generated content makes sense. So if you don't know if it makes sense, it could have generated something that is useless, something that's bad, something that won't work. So a great idea, like something that I suggest to people is ask AI to generate a recipe for bread. Something simple as a recipe for bread. Try to make that bread. Sometimes it won't work. Okay. Because anything that's bread related, it's kind of tricky.ADRIANA: It's voodoo. Yeah. Bread can be tricky, for sure.RENATA: Yeah, sometimes it won't work. Maybe it will work, but you can't be sure unless you have made bread before.ADRIANA: Yeah, that's so true. And it's interesting, too, because even if you've not made bread before, right? And then you take the AI recipe, try to make bread, it fails.Then you can use that as a springboard to like...but why did it fail? Then you can do some additional research, right? So still need to use your brain there, which I think that's at the end of the day, the important thing, right?Even the AI prompt engineer, when I first heard this idea of a prompt engineer, honestly, I thought it was funny. But it's in the same way that we, as software engineers or ops folks, whatever, SREs when we're trying to solve a technical problem, we're going on Stack Overflow, we're trying to, you know ask Google,like, figure out how to phrase the question correctly in Google, making sure that you're even asking the correct question. And I feel like when working with AI, it's a similar sort of concept.RENATA: If you think about the prompt engineer, which is someone adjusting the prompts they feed to AI to get the correct results, isn't that very similar to platform engineering, adjusting Terraform code to generate the correct results on the Cloud compared to what we used to do as a system in that we wouldgo directly into the machine. That is a hot take.ADRIANA: That is a hot take. I like it. I like it. That is a very cool way of looking at it.RENATA: Yeah. So maybe that is the new job, the prompt engineering, that's a new career path that someone will follow, probably in data engineering, and I'm excited to see what comes of it. I am very open to new tech and seeing what the world brings us.ADRIANA: Yeah, absolutely. I'm right there with you. I think prompt engineering can be very fun. Yeah. And don't be afraid of AI. I think there's some cool things that can come of it. It can really help with our jobs, and it'll be exciting to see where it takes us. I was talking to someone yesterday about AI, and I'm like, oh, could we ever find ourselves ina position where we end up with Skynet? And you always think about these things, but...RENATA: I don't think so. I love terminator.ADRIANA: I love terminator too. I always think of Terminator whenever this AI stuff's coming about.RENATA: It is human enhancer.ADRIANA: I'm like, "All hail our Evil Robot Overlords. Here we go." But I think there are some exciting times. There's some cool stuff to come out of it. AI is a human enhancer.RENATA: I like that. Yeah, it's a good approach. I don't think we are quite ready for Skynet. It's going to take, I don't know, maybe 1000 years for us to reach Skynet level of things. Yeah.ADRIANA: Yeah. Hopefully we won't get to Skynet levels. Fingers crossed. Well, we are just coming up on time, but thank you, Renata, so much for geeking out with me today. Do you have any parting words of wisdom to share with folks out there?RENATA: Yeah, well, just don't be afraid. Embrace new tech. As I usually say, stronger people build a stronger world. And peace out and geek out.ADRIANA: Thank you so much. And y'all, don't forget to subscribe. Be sure to check out the show notes for additional resources and to connect with us and with our guests on social media. Thank you so much for joining us today.RENATA: Thank you for inviting me today.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:Jennifer Moore (she/her) is a staff software engineer at InfluxData with extensive experience in software development, devops, and testing.Find our guest on:X (Twitter)MastodonLinkedInFind us on:All of our social channels are on bento.me/geekingoutAdriana’s X (Twitter)Adriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:TI BASICQuality Assurance (QA)GW BASICArgoFirestoreDatabase schemaOpenTelemetryBull Queue NodeJSOTel CollectorDatadog agentLarge Language Model (LLM)Artificial Intelligence (AI)N+1 querySkyNetSelf-Driving PlanesSelf-Driving CarsGovernment legislation around AIFediverseLetterbook SocialMastodonRubyC-sharpTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, Reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada.And geeking out with me today is Jennifer Moore. Welcome, Jennifer.JENNIFER: Yeah, hi, thank you for having me.ADRIANA: Yeah, super excited to have you on. So where are you calling from today? I am in the Dallas, Texas area. Oh, awesome. I think you're our first person from the South.JENNIFER: Yeah, there are programmers outside of San Francisco.ADRIANA: Nice. So to get warmed up, I want to get started with some lightning round questions that I like to ask all of my guests. So it's about six questions. It'll be fast and painless.Okay, first question, are you a lefty or a righty?JENNIFER: I am right-handed.ADRIANA: All right. iPhone or Android?JENNIFER: Android.ADRIANA: Mac, Linux, or Windows?JENNIFER: I guess, Windows? All of the above?ADRIANA: All right, I think you're our first Windows person. Cool. Favorite programming language.JENNIFER: C#.ADRIANA: Cool. Dev or Ops.JENNIFER: Was that Dev or Ops? Is DevOps the right answer? Because that's what I'm going to choose.ADRIANA: Yeah, no wrong answers. So DevOps works. Yeah. When I asked this one to Hazel Weekly and she said, "Yes," so that counts too. Love it. Final question is, do you prefer to consume content through video or text?JENNIFER: Oh, text usually.ADRIANA: All right, cool. Well, that was it. Short and painless. All right, so let's get into the meaty bits. And I always like to ask my guests how they got started in tech. So what was your foray into tech?JENNIFER: So tech is always what I liked as a child. I liked computers and I took some programming classes at summer camp school thing. That was a weird dynamic, but I did that a little bit and then my high school offered some programming classes. I took those and then I went to school to university for software engineering. And after I dropped out of university, I got a job in QA, moved very promptly into QA automation,ADRIANA: Oh, that's awesome. So out of curiosity, what was your first programming language?JENNIFER: My very first programming language...I don't know...If we're going back as far as I can, then like, TI Basic.ADRIANA: Oh, nice.JENNIFER: Yeah, I think the first one I spent any real time with was C.ADRIANA: Yeah, my OG language was GW Basic, but I don't recall doing anything super damaging with it. So my real real one was QBasic. Good times. Good times. Cool. Okay, so more on the QA thing. What brought you on the path of QA initially in your career?JENNIFER: Honestly, it was the job that I could get. It was 2009, so not like the best time to be job-hunting. And I had been looking for things for a while and I got an offer for QA role, so I took it.ADRIANA: Nice. I got my professional start in QA as well. It was not the QA automation stuff. It was the clickety click, fill out Excel sheets, and sit and wait for the developers to fix bugs. So, yeah, I can definitely relate. And then after a while, I kind of begged my manager, "Please let me write code!"So was QA automation kind of the natural thing for you? Because obviously, you're software-minded, so that felt like is that what led you to it from there? From like, QA?JENNIFER: Yeah. I have some experience with software. I had done some part-time programming and obviously I was going to school for software engineering and my hiring manager at the time had a manual QA process, whatever, but was looking to set up an automation, I guess function for it. And so I think she saw my showing up as an opportunity and went from there.ADRIANA: Oh, that's awesome. Yeah. I feel like QA automation is so developer-minded because we're so tired of I think developers are lazy and I see that as a perfectly awesome thing, right? We do not want to repeat things over and over and over again if there's a way that we can shortcut it through code. So I think it's so awesome that you took advantage of that opportunity and put the developer laziness to good use, right? Very cool. Now what kind of work are you doing now?JENNIFER: So now I work at Influx Data. I'm a staff software engineer on what we call the deployments team, which is a little bit of an unusual charter but basically DevOps, like platform engineering work, and helping...The thing that we most definitely are responsible for is like our CI/CD pipelines, maintaining the good health of Argo and the automation that generates our Kubernetes manifests that we're going to deploy, and things like that.As well as a lot of things around development tooling and some infrastructure work and a lot of whatever else comes up.ADRIANA: So that sounds like a pretty good breadth of responsibilities. Because usually you see like, it feels like the description of your work is like a combination of what you would see in a DevOps team these days, plus an SRE team, plus a platform team all rolled into one.JENNIFER: Yeah, and so we do have an SRE team and they take more of the infrastructure than we do, but we work very closely together because we're using a lot of the same tools and using those tools on a lot of the same things. And so that's a blurry line.ADRIANA: Yeah. Cool. And so as part of your job, do you get to be on-call?JENNIFER: I haven't been much in this role. I expect that I will eventually have some on-call responsibilities but the team is kind of in a rebuilding phase and so there's just been an understanding that we don't have enough people to support an on-call rotation and so we'll get to things like during our working hours.ADRIANA: Got you. Now, how about in previous roles? Have you had a chance? Have you been on-call in previous roles? And what was that like?JENNIFER: Yeah. So in my previous role...this was at a company called Screencastify. They make a screen recording product and I was leading the DevOps team there and I spent a lot of time on-call there. I think in particular, we were building kind of a v2 of the platform, which included migrating data from Firestore into a grown-up database with schemas, and that was going about like these data migrations do. And then we had some staffing disruptions where several people who were very senior and critical of that project resigned somewhat in protest of some management behavior and then the whole thing kind of collapsed and I was on-call to support that.ADRIANA: Oh, yikes.JENNIFER: That was a rough month.ADRIANA: Oh my goodness.JENNIFER: But it was just for a month. Um, I understand that the team is still dealing with, you know, like, the after-effects of that, but I'm not a part of it.ADRIANA: Oh my goodness. That's got to be so super stressful in that situation. How do you deal with because it takes its toll on your mental health eventually, if not right away, given not just being on-call, but stresses of changes in your team. So how did you cope during that time?JENNIFER: It I feel like I handled it pretty well. Like I was kind of, I don't know, like I had a sort of active, ongoing incident that I had to continuously respond to for a long time there. And so I kind of just had to do that. And I think having being able to do things that I could do in a self-directed way and things that were obviously important and necessary and I could just do them without having to go through a planning process that you would do for future work was actually kind of helpful for me in that regard. I could just put out fires and I didn't need to worry about the politics that had led up to that situation.ADRIANA: Right, so you're kind of shielded or at least you're able to work kind of in a little bubble to shield yourself from some of the crap so that you could focus on the task at hand.JENNIFER: Yeah. And there was stuff that I had been wanting to do and now there was this emergency and I took advantage of it and I put in a lot more tracing and monitoring and made some application changes to make the whole thing more observable in general. And that was nice. Getting to just do Observability and reliability work as its own dedicated priority was a really nice side effect of that otherwise unpleasant situation.ADRIANA: Oh, that's so cool. Yeah, it's very interesting because I think a lot of times organizations, when they're embroiled in the royal dumpster fire of production shit storms, it's like such a reactive mode. But to be able to...I think it's so cool to be able to take advantage of a shitty situation and basically say, "No, I gotta do this so that we can improve the overall reliability of the system." Is really cool because I think many organizations would sort of almost not be in support of that, in spite of the fact that that's probably exactly what they would need to do.JENNIFER: It's like unpleasant in the moment, but it is very powerful to be able to say that if you want the system to work at all, then we have to make it work reliably, because right now it just doesn't and it isn't. And that's the problem.ADRIANA: And so how did folks like management and whatnot react once did they...Did they start to see the benefits, I hope, of all the wonderful things that you did around improving reliability and Observability?JENNIFER: Um, I think so. So, like, my sort of direct management chain had been pretty on board with the notion of improving our Observability and making reliability a priority. And so I didn't have to fight very hard with my manager or my director. But then once you left the engineering organization, that was when that sort of broke down. And the rest of the organization was very...or, you know, the rest of executive team was very focused on features and deadlines and just delivering things that they could sell to customers. And they didn't view a reliable product as being on that list for some reason, which is always an extremely weird view to take to me, because if the product doesn't work, then even if you can sell it to someone, they're not going to keep paying for it. And so why? I don't know.ADRIANA: Yeah, I totally agree with you. It's interesting, though, that you I mean, it's kind of shitty that you ended up being a situation where afterwards management didn't really see the value in what you were doing. But from a hindsight perspective, it's interesting to see at the same organization with leadership changes, what happens when you have leadership that's fully supporting this idea of, "Hey, let's make sure our systems are reliable."So supporting an Observability and reliability culture versus an organization...within the same organization, it's change in leadership saying, "No, that's not our priority." It's an interesting experiment, and I'm sure aside from the obvious things that we would think, like, yeah, that's obviously not a great idea, but getting to experience firsthand what that was like, I'm sure must have been a very interesting and unique vantage point.JENNIFER: I think it was definitely interesting. I very definitely learned a lot from it. I'm still kind of, I don't know, like synthesizing what that is so it would be hard to teach those lessons to someone else but I certainly came away with a lot of experience.ADRIANA: Silver lining, then.JENNIFER: Yeah.ADRIANA: So out of curiosity, when you're talking about bringing Observability into the picture, what did you do in terms of Observability?JENNIFER: Yeah, so like in that specific case, there were a couple of things. There was like the system had a main sort of application web server that would handle the bulk of talking to clients and ingesting video and things like that, and then a task system that would do some processing of video and analysis and things like that. And I could not see at all what was happening in that task system and I really needed to to understand what was going on with the whole system. And so I basically just stopped doing other things for a few days and wrote up a OTel instrumentation for the library that powered it, which is Bull Queue. So now there is one of those and I wrote it and I did so in anger.ADRIANA: Angry coding. Awesome.JENNIFER: Yeah. And so like that like that actually that was very helpful, probably in ways that my management did not appreciate because it illuminated a lot of areas where the problems were not occurring and the problem was basically database-related.The question was like what is causing all the stress on the database? And the other thing I did was split up all of the database access into multiple different accounts so that I could actually tell the difference between whether traffic was coming from the web service or the task workers or some migration jobs or whatever else was happening. And between those two things I was able to develop a basic understanding of what the problem was.ADRIANA: And then what did you use for visualizing your observability data? Did you guys use something that was like a SaaS product or just something that was hosted internally?JENNIFER: Yeah, they had been using Datadog and so kept using it. That was a decision that was made before I joined the company and so that just was the one that we stuck with.ADRIANA: Fair enough, but it did the job well enough, I guess, with the data that you were receiving.JENNIFER: Yeah, I mean, we could definitely answer the questions that we were trying to answer once we started sending them the data that they would need to do that with.ADRIANA: That's awesome. Do you know if anyone is still taking advantage of the Observability setup that you put in place?JENNIFER: I understand that they are when I left, one of the things that I had been wanting to do or was actually starting to do was move to the OTel special Collector rather than Datadog's proprietary one so that we could experiment with different back ends and things. And it seems like that work continued because last I heard they had moved to Honeycomb, went off to Datadog.ADRIANA: Interesting, that is very cool that the stuff you put in place continued, I'm sure that you feel really great about that, to have a little legacy.JENNIFER: Yeah. I wish it was a happier legacy, but I am glad that it's helping the people who are still there's.ADRIANA: Well, switching gears a bit, I know that when we were chatting earlier, you were talking about you had some thoughts around how engineers learn. So I was wondering if you could share a little bit more on that.JENNIFER: Yeah, sure. So this is this kind of all like this thinking that I've been doing about it kind of comes out of a lot of the sort of public conversation that's happening around like AI and LLMs and such. They're used as developer tools. And I think one of the areas that doesn't get talked about enough in this regard is that the only things that those kinds of AI development tools are really good at doing are the same tasks that humans need to do in order to learn about the complex systems and programming and the systems that they work on. And so it's a little bit of technology eating its own seed corn here when we push these tools. Because it might be convenient for senior people who already have all of those knowledge and skills, but the next generation of engineers who we should be looking out for are just losing all of these opportunities to do really good basic learning work to computers that can't even really learn from it.ADRIANA: Interesting. So you're saying that these AI tools are almost like hindering how we learn as a result?JENNIFER: Um, yeah, kind of. I mean, I think that there like there is a lot of danger that those AI tools can take all of the good, you know, like all of the good learning tasks and, you know, I guess like jobs and roles that people would learn from.ADRIANA: Right. Yeah, I think even to a certain extent, using I feel like there's a fine line, right, like, of like leaning too hard into AI and then but using it as an aid, right? For example, I've had in a couple of cases where I've written something down, I need to summarize I summarize something, but it needed to be like 300 words and I'm like, at 350, I'm like, Shit. Sometimes it can be really tricky, right? So popping it into ChatGPT and saying, "Hey, can you just make this fit into 300 words?" You've put the time and effort into writing it and then ChatGPT just takes it that extra little bit to just get you across to the finish line.I feel like that's all right versus like, "Oh, Chat GPT, write me an entire story," and then you don't really have to think, research, whatever. It's kind of like this lost opportunity for learning because you're relying on it to do basically the whole thing for you and maybe you'll verify it, maybe not, I guess, depending on what kind of person is using the tool.JENNIFER: Yeah, I think that is one of the actual uses of these kinds of LLM tools that makes sense to have the author of a document use it to produce a summary that they can verify the correctness of or to do some style transfer to make it sound like a business email instead of something you wrote at 4:00pm when you're trying to just leave the office. But that's not really how it gets...That's not the limit of how people advocate that they be used. And it instead gets posed as, like, an ops tool. And people are proposing that you should do AIOps and you'll have machines that will just scale your systems for you or whatever or tell you that you're running out of database connections and jump right to an answer which may or may not be the correct one.And without letting any people go through the path of discovering what happened there. And if you're running out of connections because you have an N+1 query that is closing and opening new connections all the time, that's a very different problem than just like a runaway serverless system that is overloading your causing some sort of thundering herd problem for your database. And the AI probably can't tell you the difference. It probably doesn't know that there is a difference, but it sort of takes away the opportunity for people to do that investigation and learn what those things mean and what to do about it.ADRIANA: Yeah, that's a really good point. And I think to a certain extent, those of us who have not, quote unquote, grown up around ChatGPT, I feel like we could be...or similar tools for that matter...I feel like it's almost like because of how we grew up in tech, it provides like a guardrail for us where we're still probably more inclined to still do the research. Trust but verify, not take it for face value. But for folks who are coming up in this era of these various AI tools, I think it becomes a lot more difficult because this is like sort of they're not as encouraged to put in that extra thought or put in that bit of creativity before handing it off to the AI tool, because that's not what they've been brought up with. And I feel like that can be very dangerous then to the younger generation as a result.JENNIFER: Yeah, exactly. And so when you're like an experienced programmer and you're using ChatGPT to remind yourself what the syntax is for array map, that's a very different dynamic than inexperienced programmer who's using ChatGPT to create from whole cloth a function that will turn one array and do something to an array and give them different results.And now that...they don't know, superficially, these look like the same thing, but there's just so much experience and context that the senior person brings, and I think that we all forget that not everyone has that. And you had to develop it somehow, and it wasn't by having a computer just hand you an answer.ADRIANA: Yeah, exactly. It's those late night sessions with like StackOverflow open on different tabs, trying to figure out what is the right question that I should be asking and please let someone have had that same problem, so maybe I know what's going on.JENNIFER: For someone who hasn't even internalized what a callback is and how it works and how you should think about it, they're in such a different position, they can't even really do anything with the result other than just paste it into their editor and see if it works.ADRIANA: Yeah, it's basically like the blind programming at that point. I'm curious to see how things are going to pan out because there have been calls for legislation around blocking these AI tools to a certain extent, or limiting them. So I'm wondering how it's going to pan out. What are your thoughts?JENNIFER: I hope that something happens, because yes. I also have thoughts on the way that these tools get built and all of the harvesting of data without permission that goes into the training sets and how opaque they are and whether what the results are and how they get used and by who and on who. These are all big questions. So yes, I do think that, like, having some sort of, you know, government regulation oversight is going to be important because, you know, like, the way these AI models are built involves a lot of, like, harvesting of you know, people's work uncompensated from the internet. It's a very extractive thing. And then they get turned into these computer systems that people use to make decisions. And you can't really inspect those decisions. And people don't really understand what they're doing and how they work and then who uses them and to do what and who benefits and who suffers for that are sort of like...I don't want to say open questions, because the answer generally is going to be the same as it usually is. People with privilege will benefit from them and people with that will suffer. That's not great for us, but that is where things are going.ADRIANA: And what I find interesting about this whole thing, too, is that even some of the folks who are responsible for the creation of these technologies are sort of like, whoa, chill out. We gotta take a step back on this thing before it blows out of proportion, which I think is quite interesting.Now, I don't want to sound alarmist, but every time I see how advanced things start getting have continued getting with AI, I can't help but get Terminator vibes. I don't think it'll be quite so drastic, but I'm like, man, Skynet might not be too far away.JENNIFER: Maybe I think Skynet is the wrong thing to be concerned about, though. Well, I think it's important to note that the creators of all of these systems, what they're advocating for, is that other people should step back. They don't want governments to tell them to stop doing what they're doing. They just want governments to prevent other people from doing the same things and that's a different thing. And then you look at other AI systems that we have in the world, like self-driving cars. They get to a point where they can do simple things, fairly reliably in controlled settings, and then you unleash them on the real world and they're constantly going the wrong way down one way roads and, like, stopping for obstacles that don't exist and completely ignoring ones that do.I don't think that language models are going to be all that different. They don't have a real understanding of what's happening around them. They're just doing pattern matching and there's going to be patterns that they haven't encountered before. And what does it do in that case? Who knows?ADRIANA: And it's interesting that's for self driving cars, which can be scary enough if things go south, but then they're talking about...there's self-driving planes out there as well, which yeah, I feel like that's a whole other level of self-driving as well, which could be interesting.JENNIFER: Yeah. And like and it's easy to see how, like, the danger in like, self driving vehicles and why you would want to be careful about that. But then you turn it into language models and it's just like the algorithm that does things, but what you get is like self-approving mortgages and that's not going to be like, different. Yeah. Like that's still going to hurt people.ADRIANA: Yeah, absolutely. Wow. Damn. Yeah. The possibilities are endless, and it's not in a good way either. Cool. Well, I know we don't have too much time left, but I did also want to touch upon...you said when we were chatting earlier...you've got, like, a little project that you have that you've been working on around the Fediverse. I was wondering if you could tell us a little bit about that.JENNIFER: Yeah. So I have recently started a Fediverse project. So, like, I call it Letterbook Social. It's a Mastodon-like microblogging service. And the thing that makes it different, other than being just not Mastodon, is that I'm trying to optimize for the needs of the operators because it's in talking to people who run Mastodon servers now that I've met some of them in the last, I don't know, what is it, eight months now? That's actually not a great experience. Mastodon doesn't have very good Observability and it is hard to scale and it's hard to deploy and the admin and the operator tools and the moderator tools, for that matter, are very primitive. And so what I want to do is solve for that. I want to make it easy to set up and easy to scale and easy to understand what the system is doing and be able to oversee it as the human being running it, which I think is particularly important in this case because these things are very frequently overseen by one single person. Which is that can be very stressful, to say the least.ADRIANA: That is very cool. So are you building it on top of existing Mastodon code, or are you starting from scratch?JENNIFER: This is from scratch, since it seems like getting changes into Mastodon is sort of an uphill climb. And so I decided that since I don't particularly know or like Ruby anyway, that I'll just do my own thing. And so now I have a C# project. Get back to that lightning question, and I get to work with C#. And I've been doing a lot of that, and it's going to be a while before it's a usable thing. But it's getting to a point where in the near future I'll have something that stands up and operates and I can start exchanging messages with other services.ADRIANA: That is so cool. Really look forward to hearing more about that.JENNIFER: Yeah. Well, I'm sure I will be talking about it a lot on the Fediverse once I have something a little bit more concrete to talk about.ADRIANA: Very cool. Do you have anything on it right now, like any documentation or are you still so initial stages that it's like no, it's just the code.JENNIFER: Yeah. So I'm doing this like it's open source. I would be happy to have people help and contribute it's on GitHub. Like Letterbook is, I think, a pretty easy word to search for. And if you want a URL, there's letterbookhq.com.ADRIANA: Send me the URL and I'll include it in the show notes. Very cool.JENNIFER: I haven't had opportunity to focus on the kinds of open source project maturity that would make it easy for people to jump in and start contributing. But if somebody is feeling adventuresome, I would love to have more help, and I would be more than happy to talk through how things are structured and what contributions people can make.ADRIANA: Very awesome. So all you C# lovers out there, cool opportunity. Very cool. Well, we are coming up at time. Well, thank you so much, Jennifer, for joining today.JENNIFER: This was lots of fun.ADRIANA: Totally loved talking about AI stuff, your Observability endeavors, and your new little Fediverse project. Thank you so much for geeking out with me today. Don't forget to subscribe. Be sure to check out the show notes for additional resources and to connect with us and our guests on social media. Until next time...JENNIFER: Peace out, geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.

About our guest:Hazel Weakly (she/her/hers) spends her days working on building out teams of humans as well as the infrastructure, systems, automation, and tooling to make life better for others. She’s worked at a variety of companies, across a wide range of tech, and knows that the hardest problems to solve are the social ones. Hazel currently serves as a Director on the board of the Haskell Foundation and is fondly known as the Infrastructure Witch of Hachyderm (a popular Mastodon instance). She also created the official Haskell “setup” Github Action and helps maintain it. She enjoys traveling to speak at conferences and sharing what she’s learned with others.One of her favorite things is watching someone light up when they understand something for the first time, and a life goal of hers is to help as many people as possible experience that joy. She also loves swing dancing, both as a leader and a follower.Find our guest on:GitHubMastodonLinkedInDiscordRedditHazel's blogFind us on:All of our social channels are on bento.me/geekingoutAdriana’s TwitterAdriana’s MastodonAdriana’s LinkedInAdriana’s InstagramAdriana’s BlueskyShow Links:HaskellErlangTypescriptBlog post: So You Want to Hire for Developer ToolingSmallTalkObject-Oriented ProgrammingVisualBasic (classic)QBasicQBasic Nibbles gameQBasic Gorrila gameOpenSSLBleeding Heart (Heartbleed bug)Cost CenterAdditional Links:Video: Hacking the Pachyderm: Scaling Servers and PeopleVideo: OpenTelemetry Q&A Feat. Hazel WeaklyCatch Hazel at QCon 2023 in San FranciscoTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, Reliability, and everything in between.I'm your host, Adriana Villela, coming to you from Toronto, Canada. With me today, I have Hazel Weekly. Welcome, Hazel.HAZEL: Hey there. I'm glad to be here and I'm really looking forward to this episode. It's going to be a lot of fun.ADRIANA: Yay. Super excited. So first things first. Where are you calling from?HAZEL: So I am calling from the sunny surprisingly town of Redmond, Washington and if you were to ask me in a couple of weeks, I'm going to be closer to Seattle. Seattle. And we'll see how that goes. But yeah, I'm in Seattle.ADRIANA: Yay. Very exciting. Very exciting. You're my second West Coaster that I've spoken to for the podcast today, so I'm being outnumbered.HAZEL: I mean, West Coast is the best coast, and I'm sorry, I don't make the rules.ADRIANA: Fair enough. Fair enough. Yeah. There is something like, I think, kind of magical about the West Coast, where it's like, chill vibes at one with nature.It's a different vibe from East Coast, for sure.All right, cool. So I'm going to start with some lightning round questions.HAZEL: Awesome.ADRIANA: Prepare. Tun tun tun...Okay, first question, lefty or righty?HAZEL: I am a rightie.ADRIANA: All right.iPhone or Android?HAZEL: AndroidADRIANA: For personal use. Do you prefer Mac, Linux or Windows?HAZEL: Simultaneously, macOS and 'Nix OS.ADRIANA: All right, cool.Favorite programming language.HAZEL: Probably....I feel like I'm obligated to say Haskell because I'm on the board of directors of the Haskell Foundation and it's true that is one of my favorites.My cheeky answer is also that my favorite programming language is the one that I write in and nothing bad happens.ADRIANA: Awesome. I do like that.All right. Dev or Ops?HAZEL: YesADRIANA: Yes. Okay. Wow. Both. Okay.And final question. Do you prefer to consume content through video or blog post?HAZEL: Blog post. I read way too fast to sit there and not read it.ADRIANA: I am the same way. I was just telling my previous guest the same thing. She also prefers blog posts because I cannot sit there and just listen to someone go, "Blah, blah, blah," where I'm like, "Get to the point."HAZEL: I mean, developers being ADHD in this economy, who would have thoughtADRIANA: I know, right? For realsies. So I guess let's get into the meaty bits.Well, first things first. So before we get into what you do, tell us how you got to where you are. How did you get your start in tech?HAZEL: How I got my start in tech? That is a really interesting question because I have mostly had moments of a ridiculous amount of luck and then the ability to at the moment capitalize on that luck.So how I got my first job was I was at a programming lab at university and I ended up happening to overhear two people talking about Erlang. And I was like, "I know that language."Well, I didn't know it, but I knew of it, and what kind of weird ass nerd knows about Erlang in undergrad?So I ended up talking with them for 2 hours after the lab and one of them actually said, "Oh hey, I work at a company and soon we're going to have internships. Do you want to do an internship here?"And I was like, "Oh hell yeah, I do." Because I didn't have any other options.And so eight months later I actually ended up getting internship there after applying to bajillion other companies and none of them gave a shit about me because I hadn't even graduated yet.So it turns out that he was a racist, misogynistic, terrible person who liked to rant about like weird religious topics in the middle of a Costco food cart.But other than that, it was an interesting first experience.ADRIANA: Damn.HAZEL: But if it hadn't been for that one moment of me knowing about Erlang, I wouldn't have had that job and then I wouldn't have been able to seal up in all the random weird crap that I had to, that got me my second job.ADRIANA: And so what was so what was like? What was your second job?HAZEL: Oh, you want the full history?ADRIANA: I'm curious about the second job.HAZEL: So the first job I have finally had enough of the person being really toxic because he had actually gotten the other intern to quit.And I realized that his M.O. was going through and finding gullible college undergrad people. Getting them to be interns and then just having them be super cheap rate forever until they finally rage quit and left. And he never understood why everyone left and never talked to him again. Then I left and never talked to him again. Shocking.So the second job, I looked around and found a company that was hiring and they were hiring for a front end job in React and TypeScript and all that stuff. And I didn't know TypeScript at the time, so I took a brief six hours and learned TypeScript and then took the interview and aced the interview because I had actually ended up, out of purely coincidence, helping my father-in-law at the time get his website built for a construction contracting company.And because I had that whole thing built up from scratch, I had a huge amount of experience in that particular field that they needed.And they pulled up that website during the interview and said, "We want this."And I said, "Well, I know how to do it."So that's how that worked.And then during the first week there, I built out the entire backend in MongoDB, Node JS, TypeScript and did a whole bunch of ingestion from a very weird Microsoft Server database.That was problematic. The whole thing was problematic.It turns out that that company was a consulting company that was trying to use another company in order to bootstrap themselves without getting funding so that they could actually go and do the thing that they wanted to in the first place.So you had a whole company whose entire existence revolved around their one single clients, never figuring out why they were paying this much money for a single website.ADRIANA: Damm.HAZEL: I know.ADRIANA: Wow.HAZEL: Shockingly, that didn't turn out. I don't why.ADRIANA: What a shocker.HAZEL: So after about after about ten months of that shit, it all fell apart. But in that time, I became the senior-most engineer on the team, IC-wise, built out an entire component library, re-did all the local developer environments, rebuilt everything in Docker, did like, a 10x performance improvement on the entire website and a 10x performance improvement on load balancing.Got the entire back-end working more efficiently, leveled up the entire team in terms of being able to use the component library to redesign the entire website to meet the neurotic and weird ass requirements of a client that literally did not understand how things worked.ADRIANA: Damn. So was all this happening, like, while you were still a student?HAZEL: I, so I graduated halfway through my first job.ADRIANA: Okay.HAZEL: So I went from that first job to having designed the entire design system and done all those other things. That was about one to one and a half years after graduation.ADRIANA: Okay. Now, was area of study related to what your work was?HAZEL: My area of study was computer science and the university was Portland State University. It's a great university, but did it give me the tools that I needed to actually do the literal work? No, it gave me really good tools for understanding theory.ADRIANA: Okay.HAZEL: That didn't have a lot to do with programming. So, like my very first day on my job at my first company, I still remember it took me 4 hours to do some weird jQuery nonsense with a list in HTML.And finally the head developer person was like, "What's taking you so long?"And sat there and did the whole thing in front of me, essentially like ten minutes.And then he was like, dude, "I thought you were good." Basically.And then it was the weirdest thing was like his look on his face was, "I know you're smart, but what the fuck?"ADRIANA: Yeah, it's funny because I do find, like, school never quite prepares us for the workforce because yeah, I mean, it's too much theory, not enough practical stuff.The practical, quote unquote stuff that they make us do is so irrelevant that when you hit the workforce and you're solving the real problems of the real world, you're like, oh, shit, I have to learn this stuff from scratch.HAZEL: Speaking of things in the university world that are not relevant at all, but I had a lot of fun doing. One of my favorite things that I ever did was during the operating systems course, we had to take a toy kernel and implement a multilevel feedback process, scheduler and priority queues.So I implemented that with the vast majority of the state machine logic being implemented in about 80 to 100 lines of C preprocessor macros that were recursively, expanding using macros and a whole bunch of various extraordinary crimes.My code was beautiful. It magically scoped in variables that were hidden. It did a whole bunch of things. It relied on some undefined behavior. I had to turn on GCC pragmas so that things actually compiled because Dead Code branch elimination wasn't working with ternaries and it was glorious.Absolutely none of the TAs after the third assignment would touch my...like, none of the graders would touch my code. The only person who would grade my code was the TA who was a grad student and she's still a friend to this day. But that code was cursed.ADRIANA: That's awesome. That's awesome.HAZEL: It prepared me for TypeScript, is what I'm saying.ADRIANA: So it's funny how the little things prepare us for the things that we don't even know are in store.HAZEL: Right?ADRIANA: Yeah. I feel like my whole life has been that. So one of the things that we have chatted about previously is, well, you've got many hot takes, and I feel like...we've talked about your thoughts and feelings around platform engineering.So I was wondering if you could share that with folks because platform engineering is the hot topic of the day and everyone's got an opinion. There are tons of discussions going around. So throw your thoughts into the ring here.HAZEL: Wow. Did you allocate the full four days required for this podcast to have all of my thoughts on platform engineering?ADRIANA: No, shortly, no. I mean, sadly, noHAZEL: Sure. So I'm going to have to summarize a bit. I wrote a blog post recently called "So You Want to Hire for Developer Tooling?" And in there I talk a lot about the first platform engineer or the first people that will become platform engineers in your company and how to not fuck it up.And I'm sure a lot of people are going to read it and then fuck it up anyway, and that's fine. It's really hard to hire for it.But hot take-wise, platform engineering is something that I find really interesting in that in the industry I see this habit of over and over and over.Someone says, "Oh hey, in a sociotechnical system we need to solve the technical problems for the spice of the social problems and solve the social problems for the spice of technical problems and have them work together in a collaborative fashion, understanding the constraints and challenges of both."And then someone goes in and says, "But tooling and vendors" and the whole thing goes to shit.And this repeats over and over and over as organizations don't want to skill up in the cultural maturity and outsource understanding of something that they see as not a core driver to the business, which in modern economic theory, it makes sense.If it's not a core competency of the company, why should you not externalize it and view it as a cost center?And since the world is being eaten by tech, people haven't seemed to catch on to the fact that developer tooling, how developers work, the entire process of how they collaborate with each other and with the company is now inherently a core competency of existing in a tech-driven world.So if you want to be relevant as a company, it's not platform engineering, it's not DevOps, it's not tooling...you need to understand how people work together and how people and solutions and technology work together and how to scale that understanding.And the problem you will always run into when doing that is if your work is meaningless or if you are ruled by toxic work behaviors or you have a bunch of institutional biases and corruption in your company that prevent people from genuinely being able to improve the system as they see it, you will always end up with a broken system.And so if you talk to executive leadership and say, "Oh," and they ask you, "What can we do to improve developer productivity?"It's not productivity you want to focus on; it's the developer experience. And the developer experience there. The biggest leverage you're always going to have has nothing to do with the tooling, has nothing to do with Kubernetes, has nothing to do with fucking YAML.Although swearing has a lot to do with the YAML because it's a natural and necessary defense mechanism when you have to write it every day.With that aside, if you're going to improve the experience of developers at a company, the work has to be meaningful, the work has to be high impact. The work has to be high leverage and the relationship that the company has with the developers needs to be healthy and fulfilling and equitable.And you will find very little leadership that is willing to take the full implications of that and execute on it.ADRIANA: Yeah, I completely agree and I think that's, I mean, that's why we see so many of these so- called "transformations" fail miserably. Because leadership doesn't have their heart into it.HAZEL: Mmhmm.ADRIANA: They've been told by someone else who's like, "Hey, do this, it's in vogue."HAZEL: Yeah.ADRIANA: Go on, go forth, do it. And it's just like business as usual.HAZEL: Did you know that related to that, it turns out one of the best indicators of quality in the system is whether or not people genuinely enjoy working on it.And if you ask the question, "How does this system, how does your experience with interacting with this make you feel?"Does it make you feel more alive and more whole?If the answer is "Yes," it's probably a good quality system.And if you need to choose between what to do, you can always ask yourself, "Which of these options will make me feel more alive and whole when interacting with the system?"And a lot of people will go, "But what about the quantification?"Like, what about the numbers? This seems like hippie mumbo jumbo.And no, you should be in touch with your fucking feelings. You should be in touch with the human side of yourself, and you should not just bury it deep in your ass crack in the name of capitalism.It's literally more efficient to actually think about your feelings and think about what it means to be a human and the human experience and try and make the world a more wholesome and inclusive place for everyone around you.It's literally more efficient. It's literally objectively superior in many ways. And you can show that.ADRIANA: Yeah, that makes a lot of sense, actually, because I've definitely felt in the past, like, when I'm working on something where I feel like it's a pleasure to come into work every day because this work is fucking cool. I think you're going to see it in my code.You're definitely going to see that extra. I'll go that extra little bit. Right? Just to get it done. Because I'm excited about what I'm doing. Because I think this is cool shit that we're trying to do here.HAZEL: And you're invested in improving a system that you care about and that, you know, cares about you.ADRIANA: Exactly.HAZEL: No one wants to work on a broken tool and no one really wants to fit a broken tool if they don't think that it will actually be received. Like, you can't fit something that won't be integrated into the system and it cannot see improved something that doesn't want to be improved.ADRIANA: Yes. And it's both by design, doesn't want to be improved by design and doesn't want to be improved by the human overlords of that tool. Right? Because I've been in so many situations where you come in and they're like, "Oh, yeah, this thing's a piece of crap and it could really do with refactor or rewrite." But then no one wants to invest the time. Right?I've felt so many times where it's like this thing just straight up needs to be gutted. Like, keep somebody keep a team working on this thing to keep the lights on while the other team does the rewrite, right? So we can all be happy in the end, but a lot of organizations aren't willing to invest that extra time and money, right? Because that means you've got an overlap of like, two teams working basically on the same thing.But I feel it ends up being a very short-sighted decision to not support those types of things because you're shooting yourself in the foot in the end.HAZEL: Yeah. And so with migrations in general, one thing that's really interesting about that is it turns out there's like a pretty formulaic strategy you can use in order to execute a migration of arbitrary size and complexity. So there's three main steps that go into a migration.The first step is de-risking a migration. So that involves talking to people and understanding what they actually need and working with the people that are being hit the most by the inefficiencies and the insufficiencies of the current situation. And then you get them the new situation and you make sure that it works. You work with them, you work on that. You make sure that this thing will actually do what you want it to. That's the first step.The second step is the enablement, where you say, "Okay, what is all of the low-hanging fruit?""What is all of the automation we can do?""How can we take this migration?"And as much as possible, take the toil out of it and take it out of the hands of people who don't have the context required to execute the migration. How can we facilitate that?And the third step?The third step is literally someone needs to sit there and A commit to finishing it and B commit to communicating about it.So the thing that I find fascinating about migrations is that most migrations fail in the first stage of knowing, actually sat down and talked to the team before they ripped out a solution. Like people will rip out a solution that isn't broken or people will try and say, "Here's a new solution" that actually makes the problem worse.If you just talked to people and actually worked with them to verify that something will be in fact the solution, you would save millions of dollars a year or hundreds of millions or even billions in your company over time. And you would save years of developer effort by just fucking sitting down and talking. And it's ridiculous that this is not a thing.The second stage migration.ADRIANA: Yeah, I actually agree with you.HAZEL: And the second place the migrations fail the most is people celebrate the automation step and the majority and they don't celebrate the done done of, "We actually finished everything and turned off the old system."Don't celebrate if you haven't turned off the old system and sit there and commit to fucking doing the last mile. If you initiate the migration, it's on you to finish it. You cannot just hand that off to someone. It's on you to finish it, but it's also on you to communicate about it.And so many migrations are not able to be finished because the communication of your progress, the communication of the value, and the communication of what needs to happen in order to actually do this, never happened. So again, talking to people, or rather the lack of talking to people, kills most migrations. And it's astounding to me because sure, it's difficult to understand what leadership or what your management or other stakeholders or other teams are looking for in understanding the progress of your migration.But it turns out there is a simple and effective strategy to figuring out what they need in order to feel like you're communicating with them.You ask, "Hey, is this working for you?"And they say, "Yes," or "No."And if they say, "No," you change it and then repeat, right?ADRIANA: That's it novel concept, right?HAZEL: It's not like we've had language as a society for like 18,000 years or something, right?ADRIANA: I know, right? Yeah. But it's so true and I think that's like the most fundamental problem that we see across the board with these types of initiatives.My favorite example is always, like, infosec. I worked at a bank a gajillion years ago and we were like so we had admin mode. Like, developers had admin mode on their laptops and we were able to install certain software so that we could get the job done.And this was, I believe, it was like, pre- approved software to begin with, but then all of a sudden, InfoSec, one day they're like, "By the way, we're going to block the installation of all software unless it's on a whitelist."And then unfortunately, we had to discover as we went. Like, "Oh, shit, this is blocked."OK, well, now we need to contact InfoSec to whitelist this. And we couldn't complete the simplest tasks. I mean, it was ridiculous. All of a sudden, developer productivity went to a standstill because InfoSec didn't bother to speak with development teams to talk about, like, "Hey, what would your workflow look like if we did this? Right?So it was just like the directive came from whomever. And thou shalt live with this heinous crime against development. So yeah.HAZEL: I mean, the real solution there, the real solution there obviously was to have all the developers move to Visual Basic and Microsoft Excel as their main development platform. Because it turns out Microsoft Excel is one of the most efficient, beautiful, and glorious development platforms out there. And it's one of the best programming languages, too.ADRIANA: Oh interesting, Excel. Yeah, I guess so. Yeah.HAZEL: Honestly, it is actually a good programming language because it turns out so one of the people that says that is Simon P. Jones, who is one of the people that helped create Haskell.So actually the Visual Basic inside Excel is a functional programming language that has first-class functions. It has a whole bunch of other like, not-so-nice things in it. It even has lambda functions. It has all the fun, hot, trendy things. And the reactive programming model inside Excel was later stolen and turned into ReactJS.I'm modifying the history. I'm going to pretend it was stolen from Excel. But Excel is actually pretty awesome. Like, in terms of a programming language, it would be hard to find something that is more accessible to people outside of tech.ADRIANA: Yeah, that's true. You're talking about, like, straight up Excel formulas at this point. Are you talking about, like, okay, not the fancy stuff that you can do with, VBA?HAZEL: They're the same thing. You can stick VBA inside basically any Excel cell.ADRIANA: Very true. You know, like speaking of VBA and Visual Basic. I liked Visual Basic. That was like, I guess, officially my second programming language. I started on QBasic back in the day. Yeah, back in the day when it with the gorillas throwing banana bombs to try to destroy the city. And there's also the Nibbles game. Big fan.HAZEL: Nice. So you've always been a BASIC bitch is what you're saying?ADRIANA: I mean, BASIC was basic, but I liked Visual Basic. I thought it was intuitive. It was a nice way to develop GUIs I mean, especially when you go from Visual Basic and then try to develop any GUI stuff in Java. That was a fucking nightmare. And I did not last more than 2 seconds trying to sort that out before going, "Buh bye!"Yeah, Visual Basic was great. It was fun. I used it in high school. Like, my high school programming class was Visual Basic and built some cool stuff. I did some shitty animations with Visual Basic. It was great.HAZEL: Visual Basic is so underrated and so related to Visual Basic. A lot of the programming languages and environments of the previous decades got an incredible amount of things very right. And so one of the things that I've actually said decently often about DevOps and infrastructure-as-code and all these things, is it's really just people trying to recapture the fever dream hyper productivity of SmallTalk and the SmallTalk VM, but with an audit trail for compliance.That's it. Trying to manage a wibbly wobbly ball of state in real time, at scale, without fucking it up. But the best feedback loop and the best productivity you have basically ever really been able to get in terms of being able to dig into a system.If you've ever seen SmallTalk, honestly, it's incredible.ADRIANA: I've never seen it. I've only heard of it my dad used to code in SmallTalk back in the day. Would not shut up about it. Yeah, that was because I think it was like one of the original object oriented languages out there. Right?HAZEL: It was THE object-oriented programming languageADRIANA: There you go. Yeah.HAZEL: And every other language after SmallTalk took object-oriented programming and said, "What if we ruined it?" And then proceeded to do exactly that?ADRIANA: So basically no one has succeeded in capturing the glory of SmallTalk, is what you're saying.HAZEL: And we probably never will, because now people are really used to being able to undo something or statically analyze something. And honestly, both of those are extraordinary inventions like the ability to say, oh shit, never mind, is actually really good. However, that has a lot of false confidences, in that, in the real world, your system is actually pretty mutable and pretty ugly anyway. So for people to say, "Oh, we can just undo this," or "Our state doesn't actually REALLY exist," that is kind of untrue. And so pretending that that's the case leads to developers having this very mismatched and distant view of the system that they work with.Whereas in SmallTalk, if you fucked up production, you knew the second you hit enter, because you just crashed the entire VM and the entire company is now screaming down to its knees, sobbing, the whole thing fucked up. But you KNEW...INSTANTLY.ADRIANA: Right. Yeah. So you get that immediate feedback versus the pussyfooting around maybe there's a thing that's wrong.HAZEL: Yeah. And the immediate feedback, it makes you fear yourself the appropriate amount. Like, if you release code that's about to nuke production, "What could nuke production?" You're gonna double- check it, whereas now, we're just like, "It's stateless. It's in Kubernetes. It's totally fine. We can undo this."And then you actually delete half your database and then OpenSSL Bleeding Heart happens and then all these other things happen, and it turns out that you're just like, you're crying in a corner, you eat 20 years in five days, you're like, stress bleeding out your toes. It's a whole thing.ADRIANA: That's that's actually a really interesting way of viewing it because yeah, I agree. It's similar...This reminds me of the argument where making developers responsible for their code once it goes into production, rather than throwing it over the wall, right? Because if you're the developer responsible for your code, there's no fucking way you're going to let shitty code go into production, because you're the one who's going to be on the hook if something happens.HAZEL: Right. A lot of people will think about that and they'll go, "So if I just make everything the developer's responsibility, it's all better." And that's not true. Because, with equal power comes equal responsibility.But with equal responsibility needs to come...With greater responsibility needs to come greater agency. Agency and responsibility cannot be separated because they are the same thing. And if you pretend that they're not the same thing, you're going to end up with a bunch of pissed off, burnt-out developers who hate you, your whole company, everything about you, and they're going to burn the entire economy down to the ground.ADRIANA: Yeah, absolutely. I yeah, and I think, unfortunately, that's kind of how I felt early in my career. I was just so fucking burnt out. Like my first job out of school, I was pulling these ridiculously long hours where I was working six days a week, 14-hour days.And I remember I complained...this other guy, and I complained...like, I was fresh out of school. This other dude, Dale, he was engaged, so he's like, planning a wedding, and we're both like, "What the fuck, man? This is like, way too much work. We're dying."Like, we have no life. And so we're like, okay, we're going to complain together, right? And then Dale bailed on me and I complained to my boss and then he's like, "Oh, you can have the weekend off."And I felt so guilty. I felt so guilty for taking the weekend off because the rest of my team was like working and Dale bailed on me. So I was like the little prissy-ass bitch who was complaining about, "Oh, she can't handle the work."But as a result of that, I had zero vested interest in seeing that thing succeed because I was like, I hated that system. I'm like, if it goes down in flames, I do not care because they treat me so badly. I don't care. I don't care about my work.HAZEL: And if you were to take any of the executives and just talk to them and say, hey, you just ruined any capability that this company had of building a team that is engaged and able to actually put everything where it needs to go, they would just look confused and go, well, this is a cost center, so why do I care? But the knowledge required to operate and build and improve this is really about something that fundamentally can't be a cost center.ADRIANA: Yeah, and that's interesting because I feel like this whole, like, "You're a cost center" argument ends up really interfering with innovation and productivity because, well, it costs too much. We can't invest in that. You're not making us money. And it's to the detriment of the entire organization as a result.HAZEL: Yeah. One of the things that I always try to do when I am leading infrastructure teams is I see infrastructure as the way, or not "The Way," but as "A Way" to enable people to have low risk, high quality, rapid experimentation.You want that experimentation to be risk-free, and you want to basically sow the seeds of innovation. And the way to do that is you get a bunch of people that are smart, you put them in a room, you more or less let them do whatever the fuck they want to, as long as they have like, a vague sort of agenda that they're kind of going towards. And then you let them try out as many ideas as possible, and you let them understand the system.Because this whole idea of software development, or development, or building a platform is really about, "How do I understand this so deeply and intimately that I can express the entire understanding of this problem in a way that other people can interface with this as if it was knowledge made concrete and tangible, and interactable."And that requires you to try out a whole bunch of things that are not that thing. It requires you to evolve the understanding of that thing over time and the understanding of the knowledge itself, the process of getting that knowledge, and the process of even thinking about what it means to communicate about it.And that's what you're doing. It's not programming. It's knowledge work. It's creation of understanding itself.ADRIANA: I think that's such a cool approach, because I think by having these loosely-defined borders...parameters...it opens up your mind to creativity.Because now it's like, oh, I feel like if you let people do their thing, I think they will naturally gravitate towards finding the problems to solve and then they will be excited about solving those problems. And like you said, they'll learn things along the way. And for me, I think one of the coolest things after solving a ridiculous problem is taking a step back and thinking, holy shit, look at all the things that I learned along the way to be able to get here and having there's no better way to inject enthusiasm into a team than doing that.Personally, I always tell my bosses, "I don't like being bossed around." I thrive...And that's the thing I appreciate about my current boss is...They know that I thrive from doing my thing and doing it well, and finding cool problems to solve and then writing about it or whatever. Like sharing the knowledge in whatever way.And I think more managers need to recognize that because the field that we are in is ultimately a very creative field, contrary to popular belief.HAZEL: It's one of the most creative fields out there. And one thing that I think of, that you reminded me of is we have the concrete work of doing something, and then we have glue work, which is tying together things in a way that is not necessarily recognized. But there's a third secret option. It's not glue work, and it's not the concrete things. I'm going to call it innovation work. It is work of finding inspiration and drying it out and bringing it to life and sowing those seeds.And it's not glue work. It's not concrete work. It is the work of divining inspiration itself from sources around you and making that visible and making the process visible and figuring out what it means to be innovative and to execute on visions you don't even know you need to look at.ADRIANA: It's yeah, and sometimes that means like, finding collaborations in the periphery of what you're doing, or finding connections to your work from somewhere that you wouldn't necessarily see that connection, because everything I think, brings us to where we need to be.It's kind of like what you were saying. I think we were talking about this earlier. Career-wise. All the things that we do, all of our experience leads us to where we are now. And you draw on that experience. You draw also like what you said on the serendipity and the opportunity taking advantage of lucky situations. I mean, you're only truly lucky if you take advantage of that situation.And I think a lot of us tend to not recognize when we are in a lucky situation and that like, hey, this is something that I need to grab a hold on before it goes away.HAZEL: Yeah. And fine-tuning that notion of luck and that gut instinct of I should focus on this or I should prioritize. This is something that I've done a lot of and it's been one of my greatest career accelerators. Because that fuzzy feeling of this is important, or this person is cool, or this is where I need to be in right now, or I need to go into this room. I don't even know why sometimes, but I just trust it because it's going to lead me to pretty cool places like here.ADRIANA: Yeah, actually that's a really, really excellent point is trust your gut. Trust the fuzzy feelings.HAZEL: Unless it's talked about, then don't touch it.ADRIANA: I was just going to say we are coming up on time. But before we go, I would love to get any hot takes or words of wisdom for our viewers and listeners.HAZEL: So a hot take. Someone asked me once to explain what Kubernetes was because they felt like it might have been a practical joke or something, because they're trying to figure out what it is, and there's just so much nonsense going on.And so my explanation of what Kubernetes was...is...Kubernetes is what happens when you take about five to ten years of institutionalized tech debt, reinvent it, and create an entirely new parallel universe of tech debt as a consequence. Which is to say that it is highly effective, and yet much of it, to some degree or another, is simultaneously needed, yet unnecessary.ADRIANA: That is awesome. I think that is my favorite view of Kubernetes to date. So thank you. Awesome. Well, thank you so much, Hazel, for joining me today on Geeking Out. Y'all. Don't forget to subscribe and be sure to check out the show notes for additional resources and to connect with us and our guests on social media. Until next time.HAZEL: Peace out and geek outADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet.Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.