The One Where We Geek Out on Observability Engineering with Iris Dyrmishi

Sep 17, 2024

Ask episode

Chapters

Transcript

Episode notes

About our guest:

Iris Dyrmishi is an Observability Engineer dedicated to the belief that observability is fundamental to a company's success and the performance of its tech stack. Enthusiastic about sharing insights through speaking and writing, with a particular focus on observability and OpenTelemetry.

Find our guest on:

Find us on:

All of our social channels are on bento.me/geekingout
All of Adriana's social channels are on bento.me/adrianamvillela

Show notes:

Additional notes:

KCD Porto

Transcript:

ADRIANA: Hey, fellow geeks, welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And geeking out with me today is Iris Dyrmishi. Welcome, Iris.

IRIS: Hello, Adriana, nice to be here.

ADRIANA: So happy to have you. And Iris is one of our, I would say, like On-Call Me Maybe alum, and it's been cool to be able to like bring various folks who have been on On-Call Me Maybe onto Geeking Out. So I'm super excited to have you on here. So where are you calling from today?

IRIS: I'm calling from Porto, Portugal.

ADRIANA: Awesome. Well, are you ready for our lightning round questions?

IRIS: Yes.

ADRIANA: All right, let's do it. First off, are you a lefty or a righty?

IRIS: A righty.

ADRIANA: Okay. Do you prefer iPhone or Android?

IRIS: iPhone. I used to be an Android freak until two years ago and I switched to iPhone just to try it and now I'm obsessed.

ADRIANA: Oh, you're a convert. Woo. Welcome to team iPhone. Awesome. Do you prefer Mac? Mac? Oh my God, I can't talk. Do you prefer Mac, Linux, or Windows?

IRIS: Mac all the way. Yeah, my Mac suffers a lot with me, but it's my best buddy.

ADRIANA: I feel you. The other day I think I took my Mac to the max. I have an M1 Mac, and those ones don't have fans. They never heat up. I was working outside and we're having like a mega heat wave right now in Toronto, like a heat dome. And it's been like, I think with the humidity, it's been like feeling like 40 degrees, which is outrageous. And my Mac was actually heating up on my lap and I think it was because the outside temperature was like, it was like, yo, you gotta bring me inside. So, yeah, too much, too much. Cool.

ADRIANA: Okay, next question. What is your favorite programming language?

IRIS: I would say Go. If you asked me a year ago it would be Java, but now I'm liking Go a lot. So that's my go-to language.

ADRIANA: And Go is so compact compared to Java.

IRIS: I've suffered a lot with Java, not a lot with Go, so I highly recommend to get into it. It will make your life a lot easier and everything Observability right now it's written in Go, so it's good.

ADRIANA: There you go. So it's perfect. It's funny because like you mentioning...I suffered a lot with Java, I can definitely relate because for me, I don't know if you've ever experienced this, but like every time I set up a JVM on a new machine, it always caused me problems. Or also like, some software was using whatever version of the JVM and you're writing your stuff in some other version of the JVM and they no likie each other and...

IRIS: It's crazy.

ADRIANA: Yeah. Go is very opinionated. I do appreciate that about it. Okay, next question. Do you prefer dev or ops?

IRIS: Ops. I got trained as a dev, but I started working as an ops very early in my career and I love it. Now I'll never change again.

ADRIANA: It's funny because, like, you know, a lot of times, like, there is like, you know, in school, there's...you can either, like, get a degree in computer science, computer engineering, or like, you can go to a coding camp. And so there's training for dev, right, but there's like, no training for ops. How wild is that?

IRIS: Yeah. I'm actually thinking I want to get a master's degree. I want to further my studies now, but I'm so deep into my career, into ops and doing a master's degree, it would feel just like doing it for the sake of it. There is nothing that will further my knowledge in the ops field. It's crazy. I'm really trying to find a good program, but it's just impossible. It's either game development or back end development or for example, machine learning, which are, of course, good skills to have. But if you are doing that degree to improve what you're currently doing, it's, it's impossible. You cannot find anything with ops, especially Observability, of course, but, yeah, in general. Yeah.

ADRIANA: Yeah, I know, right? Yeah, you need like an Observability camp.

IRIS: We should make it happen.

ADRIANA: I know, right? There you go. There's. There's the idea of the day. Okay, next question. Do you prefer JSON or YAML?

IRIS: YAML. I work a lot in infrastructure with YAML, and right now I can debug it with a clear eye without even needing anything. Like, I can see. Ah, there's a problem. It is the problem with the indentation. But, you know, I'm so used to it now, how it's supposed to look that it's very. It comes very easy to me.

ADRIANA: Yeah, I was gonna say, like, being so, so heavily invested in the ops side of things. Like YAML, YAML, Go, is like part of the ops toolkit these days. Okay. Spaces or tabs?

IRIS: Spaces. I feel like I have more control over the spaces. With tab, it's like too much. With spaces, you can do one at a time and fix things.

ADRIANA: I'm with you on that. Okay, a few more questions. Do you prefer to consume content through video or text?

IRIS: Text. Because the video, I get distracted very easily. If I'm watching a video, I'm thinking a thousand other things and I will not get the knowledge that I need. By reading, I focus and I take notes. It's much easier.

ADRIANA: Yeah, I agree. I agree. I find it, I find it really irritating when, like, I'm forced to watch a video because I can't find the answer anywhere else and then I have to sit there and sit through it in like five gillion restarts because similarly, my brain starts to, like, go in all the directions and I'm like, oh, this gave me an idea for blah. No, watch the video. Totally get it. And finally, what is your superpower?

IRIS: I would say that I get things done. I'm very crafty in life and in my work. Like, if I have something that I need to do, I do it no matter what. I find a workaround and if there is none, I'm gonna find a workaround. For the workaround, I always get things done. And that's a nice skill to have, especially in ops, but in real life as well, even like, for example, to put a picture in the wall, I don't have the tools. I always find a way. It just happens. So, yeah, it's a nice superpower to have.

ADRIANA: That is a great superpower. And it's so relevant for our line of work. I mean, for any, any line of work. But I feel like for our line of work, like, the craftiness translates to creative problem solving, especially when we are hampered for whatever reason from doing the thing. So I think that's so cool.

IRIS: And makes work and life fun. It really challenges you when you have to get crafty, so, you know, you never get bored.

ADRIANA: I agree. So hopefully I don't put you on the spot asking this, but what is an example of being crafty that, like, you're super proud of?

IRIS: Actually, yeah. One thing that I'm very crafty is like, I live in a very small apartment that I'm renting right now. So I wanted to have a very fancy office set up. So I went to Amazon, I went to Google, I went to 100 different, and I bought the small pieces here and there. And I have made like three screens. Beautiful, like amazing, comfortable. You know, it's like without spending too much money. And I'm very proud of it because it was like, okay, I got this in a bargain from Amazon. I got this from there and just put it together. And without the space, it still is, like, a great place to work. And it's, like, my creative space. It's more of a home project, but, yeah, I feel proud of it.

ADRIANA: Oh, that's so cool. And, you know, like, having a cool space where you can, you know, let the creative juices flow is so important because you got to be, like, comfortable where you're working, right?

IRIS: Absolutely. And, yeah, I have my beautiful screens. You know, I look like a hacker in the movies. You know, when I was a kid, I used to watch, I was like, wow, that's so cool.

ADRIANA: It's funny because, like, you know, you mentioning, like, you. You creating, like, a nice little workspace for yourself, you know, like, thinking back to the days of working in the. In an office. Right? And I don't know if you had a similar experience, but I did go through a phase where, like, I had a nice large cubicle that I decorated and stuff, and then the company I was working at, like, moved to, like, bench seating. So it's like, you have enough space for, you know, like, your monitors, your keyboard, and maybe some extra stuff and, like, a little drawer under your desk. And it's a very sort of in, impersonal workspace at that point.

IRIS: Yeah. For me, it was always working in this open spaces that you can sit wherever you want and you have a monitor, then you can plug it in, but every time you sit somewhere else, it's never personal. So I like to have my own space to organize it how I want to have, like, a microphone here to buy little things and decorate it. It just brings pleasure. And I work fully remote now, so it's great solution to have, like, this nice little space.

ADRIANA: Yeah, yeah. And that's the thing. I think that's what's really nice about working remotely, is that you can kind of craft your own little working corner and get it just the way you like it. And I love seeing people's different setups for remote work. Like, some people, like, do really cool lights and stuff, or, like, you know, they'll do the mechanical keyboards, or they'll invest in, like, you know, three monitors, three external monitors, and it's like, oh, my God, this is so cool. Things that, you know, we wouldn't necessarily have that at an office without, like, you know, going through whatever process to, like, request extra monitors. And they'd be like, why do you need all this extra crap?

IRIS: Yeah, I bet they cannot give me a blanket that my cats can sleep next to me. That's what I have as a Christmas gift, we got a blanket from Miro, and my cats love it. So they take turns coming there. I'm working, they're purring. It's perfect stress control. I cannot get it anywhere else.

ADRIANA: And being able to work with your cats because, like, you, you hear a lot of, like, offices that are like, dog friendly and, and because of the nature of cats, I mean, I don't, I don't.

IRIS: They cannot be together.

ADRIANA: Yeah, well, that's the other thing. Yeah, they can't be together. And also, I doubt people would want to bring their cats to the office because the cat would be like, what are you doing to me?

IRIS: They like to escape as well. So, yeah, it's not a good idea. You can have them at home, but nowhere else.

ADRIANA: Yeah, yeah. Fair enough. Fair enough. Well, all right, well, thank you for answering the lightning round questions. And now we shall get into the meaty bits. And, you know, one thing that I love chatting with you about is Observability. And that's how we got connected in the first place is we got connected on Observability when we brought you on to On-Call Me Maybe, and hearing about your Observability journey on there. And it's been really cool to see you as a more active participant of the OpenTelemetry community, which has been awesome.

And getting to meet you in person at KubeCon in Paris a few months back, that was so cool. But yeah, I mean, talk about your Observability journey, how it started for you. And now I think when we talked On-Call Me Maybe you were at a different company. So this is like your second Observability role. So if you don't mind sharing your journey and how different it is, like, going, you know, like now being in your second Observability role.

IRIS: So, yeah, a lot has changed in the past year. I remember when I participated in On-Call Me Maybe I was so insecure when I was talking about it because I had been in a while in Observability, but I was still, like, building my position, my skills. And now one year later, I changed company. I'm currently working at Miro, doing Observability there and I can see how much I have evolved. Like, I have become not only good at Observability and knowing how it works, another superpower. I would say that if you wake me up at 2:00 a.m. in the morning, I usually am not very coherent when someone wakes me up. But if you ask me an Observability question, I'm gonna answer that.

ADRIANA: I love that.

IRIS: So, yeah, my passion has reached that point.

ADRIANA: Yeah.

IRIS: But, yeah, now I'm not just like a person building Observability, but I'm also advocating for it a lot. I like to think that in my team, I've advocated for a lot of good technologies of improving Observability and getting to the best possible and getting more engaged with the community. And it has been a great ride. I'm actually not just doing Observability now, but also kind of working more on architecture level to put all the pieces together. So I feel like my journey in Observability has been great, and I'm looking forward to see what is going to bring more and how it's going to advance my career. I plan to be on Observability for a very long time because I really, really like it.

ADRIANA: I love that so much. Yeah. I love your Observability advocacy because it's so infectious. And I've seen, too, that we've had you a bunch of times for end user discussion panels for you giving your feedback as an end user to the OpenTelemetry end user sig. And if I recall correctly, we also had you for OTel Q&A for the End User SIG. And even, I think we even did OTel in Practice, right?

IRIS: Oh, yeah, yeah. We were talking about Observability as a sport, I remember.

ADRIANA: Yeah, that's right. That's right. Yeah. So we're like, yeah. Because after I met you, I'm like, we must have you for the OTel End Users. So you've been such a great proponent of Observability. And I've seen also, like, got a blog on Medium as well where you, where you write about Observability. And it's so cool. I love seeing that. I love seeing the passion. What is it about Observability that, like, gets you so excited that, you know, for you is like, this is the it thing in my life.

IRIS: Well, it honestly started at something that was so new, I had never heard about it before. Like, not in school, not in work, in companies. It's like something so new. I'm like, okay, I need to learn about it. And the more I learned, I understood how important it is for a company. And it made me wonder why not many other companies have it or are building it at the time when I started my career. So I really got into it and I saw that it's like an industry that is moving so fast. It's becoming so modern, and it always has the best practices if you know how to apply them.

So it always keeps me on my feet, always wanting to improve, always wanting to learn more. Yeah, it's great. It becomes a little bit addictive wanting to know more. I stay on LinkedIn, for example, or on Medium, and I find these great articles use cases and it's just fascinating all the time. It never gets boring, basically.

ADRIANA: Yeah, I totally agree. And I think all aspects of tech Observability just keeps evolving and it's been really interesting to see it evolve over the last little while. And especially with OpenTelemetry. How would you say your experience has been with OpenTelemetry? Like when you started using it versus like now?

IRIS: So when I started, you know, it was something very new for me and of course the community had contributed a lot, but still, it was like finding the unknowns and it took a little bit of getting used to. The documentation was not, not the best. So that's why I even started my, my blog. At the moment, maybe it was just me not being able to put the pieces together, but now it has changed a lot. I see that the community is a lot more involved. We have a lot more exporters, receivers, processors, makes your job a lot easier. And I see that it's always like the maintainers are doing a great job, always keeping on top of everything. For example, we had that security vulnerability some weeks ago.

Yeah. And it was solved immediately. So it's no longer two years ago, maybe even one year ago, people were skeptical to use it because, oh, it's so new. Things are not as well. Now it is the technology to use it. So many vendors are also making it a crucial part of their solution that they're offering. So it's really become that. It's a huge transition from we're skeptical to use it to yes, let's use it, let's find someone who knows how to do it, let's find someone to instruct us how to do it properly.

And that's one of the things that I'm very proud of achieving in Miro and in Farfetch'd. Because in Farfetch when...actually a funny story because when we were in KubCon, they showed the companies that are active participants or that are using OpenTelemetry. And I saw both Farfetch's and Miro and I feel kind of proud because I was an advocate in both cases. But yeah, one year ago and now, and it's a huge difference, but it was always a good technology and you could always see the future and how much premise that it had. Yeah, I am a big fan of OpenTelemetry, and I can talk about it all day as well.

ADRIANA: One thing that I wanted to ask is, I remember, like, when you were at Farfetch, one thing that, like, really struck me was that it had this culture of Observability already, which for me was like, oh, my God, it's like Observability nirvana. Because they were, like, really wanting the team to, like, they wanted the whole organization to implement Observability practices. And I remember you saying that. I think when we chatted for Q&A that there was a directive that it wasn't, you know, developers had to instrument their own code, which near and dear to my heart. How do you compare that to where you're at, at Miro? Did you walk into a similar Observability culture? Is that something that you were kind of brought in to do to start building up that Observability culture? How did it compare?

IRIS: I think that I entered in Miro with the same Observability need and culture, but I think that my role actually, during my interview process, I actually interviewed them as well about how Observability works. Because since I'm so passionate being brought to a place that just having the Observability title and not actually doing what it is would not make sense. But, yeah, when I entered, I realized that it actually has that culture as well. They just needed more people to advocate more and to make it bigger, a bigger movement. And at the moment, we are in that stage. That Observability is one of the main initiatives in the company. Still the same. Everybody owns, instruments their code, owns their alerting, owns their dashboard.

So I'm very happy with that. I think that we're doing it correctly even here. Yeah, even now that we have more experience with the community as well. It's a great movement in Miro as well with the Observability. I'm very, very happy with that. And we're also collaborating a bit more here with other teams, for example, with performance. OpenTelemetry is helping both of us, and we're pushing it together forward. So it's a great movement.

ADRIANA: That's so cool having that culture that you're, you're walking into. And I, you know, you, you mentioned something that's so important that, you know, you interviewed them as well, because, I mean, I've always been a huge proponent, proponent of the philosophy that, you know, when you're interviewing a job, it's not just them interviewing you it's you interviewing them because you need to make sure it's a good fit for you as well. Right? Because there's nothing worse than walking into, you know, a complete shit show, unbeknownst to you, because you didn't ask the right questions. And I, and making sure that you knew what kind of work you wanted to do and making sure that you could continue doing that work, I think is really, really important. And I think career wise, we all deserve to find our little corner where we can be happy with our jobs. Yeah, I can't underscore that enough. It's so awesome that you ask those questions. Now, in terms of the Observability practice, what is the main functionality of your team?

IRIS: So my team, we are currently having a fully open source Observability platform. We have built the logging pipeline. Tracing pipeline metrics, pipeline visualization. We usually use Grafana, open source kibana. So basically we build everything from scratch. And of course we help teams for alerting to build. Alerting to build their instrumentation. Advice on best practices.

Usually we don't touch the code. I personally, I haven't done backend coding in so long that even if I wanted, I couldn't go and just like in a matter of days, get into it and help instrument. I wouldn't, I wouldn't be able to. But, yeah, usually in this part, we're advisory and just maintaining the main stack, improving it, making it better in general. Now we're actually moving, making the big move to OpenTelemetry. We finished with tracing. We're working with every everything else. So, yeah, it's basically always evolving from one place to the other to provide the best tools.

ADRIANA: Nice. And how has it gone in terms of getting people into using OpenTelemetry? Was it something...because, I mean, it's already like, it sounds like an Observability centered organization. However, like, what were they using before for before OpenTelemetry.

IRIS: So for the main reason why we went into the OpenTelemetry or how we were able to sell it, let's say, was tracing. We were using Jaeger and tracing those pillars. That was kind of the forgotten child. I went there and I was like, tracing, tracing, tracing, talking about it all the time. It's actually a running joke right now in the team as well. They're like, yeah, yeah, tracing. Yes, it's tracing. So we have now he's our senior manager, but he used to be a staff engineer in the team still working actively with us.

He said, okay, let's push it forward. Let's have OpenTelemetry and tracing become the pillar that people didn't really care about. They saw that when we had OpenTelemetry, we could handle a lot more. Some change their instrumentation and they could see a lot more information during incidents. So it was kind of selling it. By showing what a good thing tracing was and how OpenTelemetry helped, it became easier to say, especially to upper management, that, hey, this is a great tool. See what we can do with this. And for the engineers, actually, it was very easy.

Once they saw how much of a potential tracing was, they understood that other pillars will be equally useful. So, yeah, it's now it's our main, main tool that we are planning to use for our Observability needs. So it's very, very good. And when I joined in November in the team, there were some small talks about OpenTelemetry, but we were saying, oh, maybe later, maybe later. And then we were like, tracing is good, OpenTelemetry is great. Sharing articles every day about something that happened in the community. And in January, we had already migrated traces, so in two months we already managed to turn some mindset around.

ADRIANA: Oh, that's so amazing. How cool. And so I guess people didn't have too much trouble implementing, like instrumenting their code using OpenTelemetry. Like, was there any, was there any education on your part or your team's part where you had to kind of direct them, as you said, not instrument their code for them, but explain, like, this is how you approach it, these are your best practices?

IRIS: Yeah, we're doing it constantly, and we're really taking advantage of the OpenTelemetry instrumentation, the libraries that already are and the documents that are already there. Usually the team did most of the work, but we do have, for example, a monolith. And some, of course, applications are very sensitive because we have a lot of users that use our product live 24 hours, depending on where they are on. So it was a bit sensitive. So I would say that, yes, we've done some instructions and sharing documentations and inspiring mostly, but yeah, the engineers, the backend engineers have done their work in instrumenting. We're still not 100% there. There are some applications that have been instrumented, some not, but the mindset have shifted and everyone want to do it, but the priorities are different for everyone. So that's why it's good to have support from management always.

So they're the ones that are pushing this forward, where you cannot be always going to someone and telling, please, instrument, instruments from instrument, you know, it needs to be like a bigger movement that comes from a bit higher than us.

ADRIANA: Yeah, yeah, yeah. And I think, I think that's really key. I mean, yes, the individual contributors are the ones who are going to do the work, but if they don't feel like their support from up above, what's in it for me, like, versus, you're gonna do this.

IRIS: Yeah, because we have roadmaps plans that we need to follow. We can just be like, oh, the Observability wants this. Let's do it.

ADRIANA: Yeah, yeah.

IRIS: It needs to go through the right channels.

ADRIANA: So are you finding...

IRIS: Sorry, no, I said, especially in big companies, that is like a lot of hierarchy.

ADRIANA: Yeah, that's so true. That's so true. And, yeah, and that's really important to keep in mind as well. The thing I was going to ask you is, are you noticing like benefits already from the instrumentation?

IRIS: Yes, we could see, we have some applications that have spans with 40,000 traces, with 40,000 spans, for example, and they get the level of information that they get now, it's a lot more detailed and troubleshooting is a lot easier. You can already see the issue. So because we are also testing our grounds with different vendors and how the information can be sent there and can be shown to our engineers, because of course, the open source backends can do a lot, but can only do so much. And yeah, we've shown a lot of value from the OpenTelemetry instrumentation together with the help from vendors and their support, obviously. Yeah, it's been amazing. You can really see a difference on the amount of information that is being shown and how easy it has been to troubleshoot to the point that we've been using OpenSearch as a backend right now, while we still have OpenTelemetry and even dashboards were built on spans because it has more information than metrics at some point.

ADRIANA: Yeah, yeah.

IRIS: So, yeah, in a way or another, good or bad, it has provided a lot more than we had before.

ADRIANA: That's awesome. And, you know, when you're...because that one thing that I get from a lot of folks is some people get like, very overwhelmed by the sheer amount of data. So how do you, like, how can you tell when, when there's an issue, how are you able to narrow in on the actual issue amongst the sea of spans? Because, like, you have the information, but how do you know where to look?

IRIS: Yeah, it's usually seeing the status codes of the span, seeing the duration. That's very manual, to be honest, until now, especially with, we're still using Jaeger UI and it doesn't provide a lot of, for example, you can search by tags, you can see where the error happened or which of the spans had the biggest duration, and maybe it will pinpoint you the right direction. But yeah, we're looking at solutions that are a little bit more better. For example, some architecture overviews, let's say, of the application and the spans, which is more obvious rather than you going and scrolling it just shows it right there. We're focusing mostly because the visualization layer is only so much that we can do in terms of open source. So we want to kind of leverage another tool to do that for us. So we're focusing on getting, transporting the right data there so it is shown properly and it makes it easier for our engineers to troubleshoot.

ADRIANA: Cool. So then that means again, you're taking advantage of like the OpenTelemetry superpower through the Collector, where you can send the same data to multiple sources, right?

IRIS: It's a lifesaver. If anyone is listening to this and they're wondering if they should use OpenTelemetry, it's amazing. You don't even have to touch your current architecture and you can leave it running on production and you can have a full separate better one running on the side just by using OpenTelemetry. And at the same time you are building this amazing architecture and you can decommission the old one without your users even noticing anything. It's amazing.

ADRIANA: Yeah, that's awesome. I love that so much about OpenTelemetry is the flexibility. Another thing that I want to ask you about is Collector usage. How do you end up having a bunch of different Collectors? If you're able to talk about that, what's the Collector setup that you work with?

IRIS: Well, I can talk about it. We're currently, because our metrics and logging are still not there yet. We're still working on it. But yeah, we already have a deployment, but we're already deploying a daemon set on all our Kubernetes clusters to collect all our information and probably running another deployment as well. The plan is always to use because of the amount of data. It's a very big company. We even want to use one Collector per Observability pillar. One for tracing, one for logging, one for metrics.

It will just make it easier for us to know where the issue is and it will not be a single point of failure, for example, if something happens, because it could, we don't want all our signals to be down. So it's good to keep them separate. So yeah, we're leveraging everything, deployments, daemon sets, everything that's in there.

ADRIANA: Awesome. Yeah, and I agree. I think that's great that you're doing like different Collectors per signal because exactly of what you said, like you. And also I think it makes it a little bit easier to kind of like manage the data, you know, so you can isolate problems if you run into problems. And so like it sounds then like you're running most of your Collectors out of Kubernetes. Are you using the OTel Operator for that?

IRIS: Not yet. It's in the plans, but not yet. I'm a big fan of the OTel Operator. Currently we're just using the Collector, the normal Collector.

ADRIANA: Oh, okay. Gotcha, gotcha. Yeah, I find the Operator very exciting and I remember discovering it by accident and I'm a huge fan and I try to contribute to documentation around the Operator whenever I learn something new.

IRIS: Yeah, yeah. In my mind and I think in our plans, I think we in the team are in sync when it comes to this. We want to use everything that OpenTelemetry has offered and whatever it is building, including the Operator, auto instrumentation, which is great and makes life easier for everyone. So yeah, it's a process, of course, because it's a big company and things that will be slowly. But yeah, we're going to use all of it and we're preparing for all of it. Yeah.

ADRIANA: And actually you mentioned auto instrumentation. What it is, are you contending with like, are there multiple languages for like the applications at your company and if so, then like are you taking advantage? Are the languages available, the ones that have like auto instrumentation, like built in.

IRIS: The main languages that we're using? Yes, there are some corner cases that are not, but we're already preparing to create our own instrumentation there, maybe even contributed to open source or if we think that it's not good enough, just have it still use auto instrumentation, but with our own library, the in house.

ADRIANA: Cool. That's awesome. And for the auto instrumentation, because one piece of feedback that I've heard from some folks who are using auto instrumentation is usually around, like sometimes it spews out too much information. Has that been the case for like, is that something that you've experienced or you, are you satisfied so far with the amount of data?

IRIS: Yeah, currently we're preparing for it, but we're not using. But I did use it in my previous company and we were very satisfied with the amount of data. We didn't really have issues. We did see that, for example, some information was collected twice, once by our current infrastructure that we hadn't decommissioned yet, and once by the OpenTelemetry Collector. So it became a bit overwhelming. But that wasn't really the, the library's fault. It was us trying to figure out how the data, like, what to decommission and whatnot without causing any incidents to our consumers. But yes, so far from my experience with it, I haven't had any issues in the amount of data.

ADRIANA: And what about. Because you said you're mostly focused on traces, so I'm assuming there's like some plans to bring in metrics, I would assume metrics. Next store. Is it logs? Are you planning on logs at all?

IRIS: Like, we're actually doing both. We're doing everything at the same time. So, yeah, the whole team is actually. Yeah, we are in a big movement. I'm very proud to say that, that our team is like, in the movement to modernize and to use the latest technologies and OpenTelemetry is it. So we are putting a lot of strength and a lot of manpower into it, doing investigations, thinking what to do, how to roll it out. It's a movement.

ADRIANA: And have you found yourself in the position where, like, you know, you need some guidance from folks in OTel on how to implement this? Or like, found an issue with an implementation, like, have you, and if so, what have you done to resolve that?

IRIS: Yeah, I've had issues, actually. I think it was a few weeks ago, I had some issues that I couldn't find the solution of, but I just searched on Google. That's my first place. I only searched on Google. And I think I went on a forum, an OpenTelemetry forum. Somebody else had had the issue and it was resolved. And it usually, I'm up to date because I am usually on the groups of, in Slack channels in OTel Collector.

So I read everything that happens there and I see all the errors. So if it's something interesting, it usually gets stuck in my mind. And if it happens to me, I'm like, wait, I've seen this. I know that it's. Yeah, it's so many users right now that it's very difficult to find issues that nobody else has had before that you are not able to. So that's another thing that I like very much right now. It makes your job easier, especially when you are trying to work fast and not spend like days and days investigating something. There's always an answer out there.

ADRIANA: Yeah, that's so true. I had a similar experience last week where I was trying to update some stuff in the Otel operator, there's a new version of for the Collector CR and, but there was like no documentation around it. So I'm like, and I'm like, I was trying to convert my YAML right to this, to this updated format and it was not working. And I'm like googling all over the place. I'm like, ah, crap, I'm gonna have to start. I'm gonna have to post a question on the operator channel. And like, and folks on the operator channel are super nice. So, you know, it's not that.

It was more like I didn't want to waste their time on something that like, has already been asked before. So I'm like, as a last ditch effort, I started searching their slack for like my one particular keyword and I'm like, oh my God. And I found someone had opened like a GitHub issue on that. I'm like, oh my God. I have the example I'm looking for. Thank goodness. And I was so happy. I'm like, yeah, I mean, the slack channels, honestly, like, there's so much info on there. It's great, it's great. Yeah.

IRIS: I love the community. OpenTelemetry community. I've never seen anything rude happening. Maybe there is very good admins as well, I don't know, but it's always very helpful. It's a great community to be in because I have been in other communities as well. And sometimes you're kind of afraid to post and ask because you will get judged. It doesn't, I haven't seen that happen here.

ADRIANA: Yeah, I agree. I'm scared every time I post a question. But everyone is always so nice regardless. Like, train my brain to like, stop. Chill out, man.

IRIS: Yeah.

ADRIANA: I've never like, on a pull request, I've never had anybody be absolutely nasty to me. Like, it's always very like polite things, even when like, you know, I completely misunderstand the concept. It's like, well, let me clarify for you, which is super nice, right?

IRIS: I had a situation actually a few weeks ago, a few months, honestly, I say weeks, but it could be a few months that I was just looking for tasks to contribute in documentation. So I said, I wrote that, okay, I'm going to do this. But at the moment I was busy, so I didn't have time. And someone else, the admin technologies and someone else posted the MR and they're like, okay, I did it now. And they actually tagged me and they said, Iris, are you comfortable with moving through with the, with their MR? You were the one who posted first. And we need to respect that you were actually volunteering and that was so nice. I know it was not with malicious intent by the other person that did the MR. Probably they saw the task and they did it without following the instructions.

But it was very nice by the admins to just check with me to make sure that things are done properly. That made me very, very happy. I'm like, okay, yeah, this is a great community to be in.

ADRIANA: Oh, that's such a nice story. I love that so much. And you know, like especially with the docs folks, they're so nice like, you know, it's very like because they have to walk a fine line, right. Of like making sure that you don't post anything that's vendor specific. So like they've, they've got to find, follow all these rules and ensure that you're following these rules and make sure that it's a respectful community. So I really appreciate, you know, the docs, maintainers do such a nice job of that generally of I've never seen, I've never had a negative experience and it's so nice to hear a story like that as well. I wanted to ask, because you're mentioning like contributing to docs, is there any other area where you've contributed to OpenTelemetry?

IRIS: No, I'm actually working on a project. I have started it for months now and I, because currently the Kafka receiver is for logging. It is only accepting the OTLP format. So I want to, I started to work with it maybe to make it compatible with some other formats because of personal reasons, personal professional that I need to do at my work. And we really wanted to make use of that, but it was impossible to have OTLP logging everywhere. So yeah, I'm working with that. But I haven't really made a lot of progress. Yeah, I'm a little bit slow on the contribution stage.

I like, I tell myself to feel better. Okay, you write blogs. It's okay, it's okay. It's some kind of contribution. You speak about it, some kind of contribution. But it's my goal that I want to be a very active contributor because I getting so much from the community and from work that other people are doing. So I'd like to give something back as well. So that's why I'm like practicing my goal skills to make good contributions.

So yeah, hopefully soon.

ADRIANA: Oh, it's so exciting. Yay. Yeah, that's great. And I think that's a really important point to underscore too because I think because OpenTelemetry has the backing of most of the major Observability vendors. It's kind of assumed that it means that those folks will be contributing. And I think a lot of vendors have dedicated teams that work on Otel because it's in their vested interest and it's in the community's best vested interest. But then there's the other side too, which is like the end users making contributions. And I think that's an important story to tell as well because, you know, ultimately the end users are the ones who are, who are using OpenTelemetry.

And so to have those contributions and making sure that there's a path in your organization to make those contributions as well is so super important because, you know, it sounds to me like there's no issue in like Miro letting you contribute to OpenTelemetry. But I know also like in, in some companies, like even just allowing developers contribute to open source is such a difficult process. Right.

IRIS: That's very interesting actually. I've never come across a situation like this because open source is open source. But yeah, I can imagine that there is cases like that.

ADRIANA: Yeah, I think like the very corporate places like banks and stuff can be very, very protective of open source contributions. Not because like they don't dig it. I think it's more from like security concerns and so whatever, whatever concerns that they have around that. So it's just nice, and it's, it's nice working in a place where that obviously security concerns are concerns for everyone, but it's nice to work at a place where there are low barriers for contributing to open source.

IRIS: Yeah. The way I see it, if you are using open source, then you should be allowed to contribute to open source. Because even for example, if you build, let's say an OpenTelemetry receiver, something new that hasn't been done before and you want to contribute it to the community. I don't, I really don't see how that could be a concern to just keep it for, for yourself because you are already using code that is public. You know, this is going to be public as well. I don't understand.

ADRIANA: But yeah, yeah, I agree with you and that's a really important point. Like you're using the open source like you're benefiting from other people's work. And so, you know, you should, I'm not saying like everybody has to contribute to open source, but at least make it like if you're an organization and you are benefiting from open source, don't make it such a huge barrier for contributing, to allow your, your employees to contribute back to the tools that they are taking advantage of.

IRIS: Absolutely.

ADRIANA: Yeah, I know we are coming up on time. So as we wrap up, I wanted to ask if there are any words of wisdom or hot takes that you would like to share with our audience.

IRIS: Well, I would like to speak to all the engineers in the companies that they should be a little bit nicer to their Observability engineers.

ADRIANA: I love it.

IRIS: We want to collect all the data, but unfortunately, it's very expensive. It's very difficult to process it. Also, sometimes we have to make decisions to collect some, to drop some, and to put guidelines in place. Trust me, we want to collect everything, but we just cannot. So be nice to your Observability engineers and cooperate, and you're going to build an amazing Observability platform.

ADRIANA: That's awesome. And that's a really excellent point. Remember the humans behind the work that is being done. It's not just magic. It feels like magic, but it's not. That is super awesome. Well, thank you so much, Iris, for geeking out with me today. Y'all don't forget to subscribe and be sure to check the show notes for additional resources and to connect with us and our guests on social media. Until next time...

IRIS: Peace out and geek out.

ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingouthe.