While you have this abstraction. And I think that fits really well when you're talking about how teams function, right? You have a team with a mission and a charter, and your manager is like the nervous system that connects you to other teams and charters. So I feel like with a creative profession like engineering, it's actually really unhealthy to put a lot of a lot of hierarchy and like authority onto this this role when it's not one, you can kind of like envision the org chart is like flipped over. It's more like if you're a manager, it's your job support. You know, I've just been talking, talking. I can keep talking about this without stopping. So I'll pause for a breath. You're right. Everyone should care about management, whether or not they are one or plan on being one.

go in on this. But one thing that you said there that I really want to call out as like an easy tidbit for folks to take away today, you look down for function and you look up for meaning. I think that'll resonate with a lot of people. Isn't

that marvelous? I came to this topic when a friend of mine had this terrible experience as an engineering leader. Like he was hired as a VP to join this company and had like 30 engineers under him and No managers and and his boss was like, well, why we don't need managers. We're a startup. Everyone can report to you Nobody needs, you know, and he was just like oh It was like this existential crisis. Like how do you are about something that's so fundamental and that's why I wrote that blog post about Why do people need managers and there's a really complicated answer that goes into System 3 and all this stuff. And then there's a really simple answer, which is that everybody uses engineering managers. It's a pattern that we figured out that works. You can hire people who know what to do with managers. You can read books and training. Like, it's like, why would you reinvent something's profound and complicated if you don't have to? If it's not your core function as a company, you should use boring technology and boring social systems.

I think the concept of looking up for meaning also says a lot about how individual contributors can work with their managers. I think what you're saying there too is managers should be a support system for the engineers who are doing a lot of the functional work. A lot of the job of management should be making sure that the engineers have the right opportunities to grow, but also connecting that to the business, that you're working on the right things that are going to help the business grow. I think there's a lot in that both for managers, but also for individual reports and how do you kind of manage up.

Yeah, you know, when I was young, like it wasn't cool to care about the business, we're all like, oh, we care about just the technology. But like, what is it that, you know, Daniel Pink said that like, what we all went out of our work is autonomy, mastery and meaning. And what does your work mean of people don't use it? Right? What does your work mean if it's not helping the company grow? And I feel like this is something that we're growing up as an industry and realizing that, oh, it's really cool to care about the business.

Yeah. And I think that touches on a topic that I totally did not expect to go into today. But I think finding meaning, you your work in some way, shape, or form, hopefully your managers can help you do that, but if you're not doing that, I think that just leads directly to burnout. So it's very important. Oh, yeah. Oh

my God. The times in my life that I have gotten close to burnout have not been the times when I was working hardest. They have been some of the times when I was working least. But like, what unified those experiences was that I didn't feel like what I was working on really made a difference. Or it shouldn't need to be done, right? Or something like, or like, people were like, Oh, you're amazing. And I'm like, I'm working an hour a day this is not okay you know those are the times when I was like does this industry for me I mean the thing about working intensely is that some of the best times in my life at work have been some of the hardest times where I've worked the most but it's because I was so invested in what was going on and I feel like the thing about burnout and this kind of goes back to a hierarchy and authority is if the motivation is coming from someone else telling you what you have to do, it's burnout city. But if it's coming from you and you're like, I'm invested, this is part of me, I care about this, then it's such a good sign, right? Yes,

you need to pace yourself, yes, you need to take breaks, but like you're on fire, you know? And that is such a core concept that impacts everyone in engineering positions today. So maybe that is a good place to ask for advice on if there are folks out there in the audience today who are engineering managers or ICs, what advice do you have for perhaps engineering managers and helping folks find a way to feel some ownership or feel kind of more invested in their work? You

know, it's hard to give this advice kind of unilaterally because it changes so much. Like, you know, the first five to seven years of your career, you're just going to be doing what other people tell you a lot and that's okay. They have your best interest at heart. It takes that long to forge grownup software engineer. We're an apprenticeship industry, but after that, yeah, you need to care about what you're doing. I guess the biggest piece of advice that I would give all technical people is, the role of management is changing really fast. And the field of engineering is changing really fast. And I think that it it's not a good thing to get too attached to your identity as a manager or an engineer. Because I think the strongest engineering managers are never more than four or five years away from writing code themselves. They need to go back to the well and freshen up those skills. And I think the strongest super senior ICs that I've met have all been ones who spent time in in management, right? Because so many of these skills are socio technical, like they're grounded in technology, but they build on top of that with, you know, people and human and organization and business skills, thinking of yourself as a technologist, is I think the key to a really long and fulfilling career. I see too many managers who like, especially when you're young, you get promoted quote -unquote to manager and it's so flattering because you spent your whole life like looking up to these people and feeling like you're being told what to do and finally it's your turn to tell people what to do and then it doesn't actually work out. But they become managers really early and they get really attached to that and the industry changes right? It's really hard to have a 10, 20 year career as an engineering manager, without becoming one of those people who just so detached from what's going on, that you can't do your best for people, and that they don't respect you as much as they would, you know, someone who really lives and breathes and walks their problems. So yeah, that's my best advice. Think of yourself as a technologist. Think of your career as a long one. The best opportunities in tech have always come opportunistically for me. They're rarely the ones where I'm like, I want to be that in 10 years. It's like, you just, you try to preserve optionality, right? And you try and put yourself in interesting places with interesting

people. And who knows what's going to happen. I love that. maybe that's a good transition then into talking about observability. So you've been involved in the whole concept of observability as it's kind of evolved in the modern era. I bet that was one of those opportunistic moments for you, wasn't it? I did, you know, and I

was laughing because I kind of specialize in rage driven development. I mean, I hate databases and I've always ended up being the one running the databases. I hate monitoring. I hate graphs. I hate all this stuff. And somehow these are the two books I've ended up writing as databases and observability. But like hate is just another form of passion, right? It's like apathy is I guess the opposite of love or hate. Love and hate are very tightly entwined. But yeah, observability wasn't really on the scene when Christina and I started Honeycomb in 2016. And in fact, you know, we borrow the term from control system theory, where the definition of observability is how can you understand the inner workings of a system just by observing its outputs, you know, we read them, we were like, oh, classic light bulb moment, just like, oh my god, this is what we're trying to do. And it's been interesting. You know, for the first few years, we were like, okay, here's observability. And you have to have high cardinality and high dimensionality and exploreability and wide events and all this stuff. And then around 2019 2020, it started getting some attention. Then all of a sudden, everybody in the world is like, we do observability too. And so now, all of your monitoring and logging and tracing and APM and ROM, everybody's like, we do observability, which kind of threw us for a loop. These days, I guess I like to think of observability as a property of complex systems, just like maintainability, reliability, performance, which puts the emphasis on the system rather than the tools. And I really like that. But it leaves kind of it kind of begs a big question, which is that there's a really big like step function in difference, like power, ease of use, cost, everything between what I think of as the observability 1 .0 world and the 2 .0 world. And 1 .0 world is when you've got three pillars, right? You've got metrics, logs, and traces. And you're probably paying to store your data in a run tool and a profiling tool and a logging tool and a metrics tool and a dashboarding tool and all these different tools and you, the engineer, sits in the middle and what are you doing? Well, you're eyeballing shapes on graphs and going, that's probably the same as that or you're copy -pasting IDs around. You're like, well, this log line, I hope I captured this trace and you paste it in the... And it's expensive. It's unwieldy. People can get really good at it, but I feel like this model's kind of come to the end of its natural life cycle. Observability 2 .0 tools are ones where you have a single source of truth. Arbitrarily wide, structured data blobs with no indexing, no schemas, no having to predefine metrics, it's just anything, any data type you might want, you just throw it in. And you emit these wide structured log events either like one per request per service or one per spam and then you put them in a back end that lets you explore so it's a lot more like a business intelligence tool where you You can slice and dice and you can zoom in and you can zoom out and you can because data is made valuable by context right? And the problem with the 1 .0 world is that all your context is scattered around and you can't connect one piece to another. And in the 2 .0 world, it's all connected. So it's just a data type, right? You don't have to worry about, oh, is this tag too much cardinality? Oh, is this number a, I got you, just throw it in and then you just use it, just as a data struct. And it's so simple. What are we supposed to do for breaking changes? Like the semantic versioning says, you should only do a breaking change when it's like a backwards and compatible, you know, breaking change. And I think the one source of truth versus many sources of truth is definitely a breaking change. But like, it means that your cost doesn't go up, you know, seven x your request traffic, it means your cost goes up as you get more value out of your tool. Like, that's one of the biggest problems today is cost go up, the value that people are getting out of the tools is going down. Yeah, so it's changed a lot. But like, I think what we're starting see a movement picking up steam where it's like, okay, the metric has been the source of truth that we've been building tools on for 40 years now. It was built for a world when hardware was incredibly expensive, and there wasn't a lot of data. And the metric is a powerful tool for summarizing vast quantities of data doesn't help you explore your systems. It doesn't help you understand your systems or your code at all.

I love the idea of having the system kind of define what observability means. I don't know if that's a good way of saying it. But I'm actually thinking of this in terms of systems that are not technology systems. The first things that came to mind actually, for me were like banking. Honestly, like trying to understand the different accounts that I have and how much is in these accounts and how are they invested and all of those kinds of things. That is something that I really want observability on. I'm actually working on a project right now to create an app to, I hadn't thought about it this way, but to implement observability in my closet. Because I have too many clothes and I don't know what clothes I have. And in both of these real world cases, what I'm trying to do is like answer questions about a system. And that's kind of what you're trying to do here too in the technology observability sense.

absolutely. You know, there are so many socio -technical like sort of ripple effects and ramifications that come from this one small thing. Is it one source of truth or many? But one of my favorites is that historically in an observably one point of world, the debuggers of last resort, the people who understand the system best are always the people who have been there the longest. They've been through the most outages. They remember writing the most code, you know. And so, and it's kind of depressing, I think, to join a team and know that you can never catch up with the people who know it best. And that's just not true in an observability 2 .0 world. In an observability 2 .0 world, you don't have to make all these intuitive leaps and guesses and memories of past outages. The data is in the tool. The context is there. You can ask the questions you want to know. And so the best debuggers are the people who are the most curious. The people who look at their changes every day, the people who are used to using the tools and used to kind of like taking a couple of minutes, the end of the lunch break and teasing apart what's going on. Like, honestly, so like, I am not an early adopter. When I was at Facebook, I was firmly in the grumpy old man lane, you know, because I knew how to use our Nagios and ganglia tools. Like I was a whiz at them. And it wasn't until a couple of our engineers started feeding some data sets into a tool there called scuba. And the whippersnappers started debugging problems faster than I could. I didn't think it was possible. I'm like, how are you doing this? I was like, professionally humiliated. But it's really exciting. It's really exciting when people who are new to the code base could pick it up and see things and understand things, that even the people who have been working on it for 10 years couldn't see or understand using the old tools. I don't

know how you phrased it earlier, but you had a short little tidbit about context really being kind of the value in observability.

Data is made valuable by context, right? Data on an island is not

valuable. The more connected

it is to other things, the more valuable it is.

That, I feel like, is really at the core of all of this. So it's not just about having all of the data, like you're saying. The problem with traditional observability tools is that you have all the data, and the data just keeps getting bigger and bigger and bigger, but you can't understand it because you don't have the context. So what you're aiming for is a way to have all of the data with its context so that you can answer questions more easily and develop that understanding even if you don't have it

innately. Most of our tools are built in the metric, and the metric is literally a number. All of its context was stripped away at right time. You can never get it back, right? That's why you're over here eyeballing the shape on this dashboard and that shape on that dashboard going, they're probably the same ones. Nothing connects them. You have no way of answering that. You threw away that data when you gathered it. You're like, this metric is over here. That metric is over there. Never the twain shall meet, you know? You can only guess. And it's just wild to me that it's 2024 and we're still like storing all this incredibly expensive data that doesn't have any context. Yes,

I actually was just having a conversation with some engineers who work on Kubernetes, open source mainly, about how do you do that? Like, Kubernetes logs are at so many different levels because you've got the application, you've got the container, you've got the system of Kubernetes itself. And so you have all of this complexity and you have logs at the different levels. And if you have something that goes wrong and you get the error at the top level, it often doesn't tell you anything. And you have to dive down into the deeper levels to find it. So having that connection of like, this is where that came from and having more as kind of how you can actually debug things.

Exactly.

Cool. So this is what you're aiming for with Honeycomb. Can you tell me a little bit about how Honeycomb addresses this challenge? Yeah,

Honeycomb is built to spec for what we call observability 2 .0. You know, we're big fans of open telemetry, but honestly, you know, we're agnostic about how the data gets bundled and shows up. When it shows up in our APIs, we accept structured data. We recommend that people, I mean, tracing is, I think tracing has gotten a bad rap because the first few generations of tracing tools were so hostile to users. Let's put it that way. Every place I know of that rolled out a tracing tool two or three years later, you don't have a team where every engineer uses the tracing tool. You have a tool where the people who set up the tracing tool use the tracing tool. And the engineers who do traces go to those people and ask them for help when they need a trace. Like they're just absurdly like walled off and difficult. But Honeycomb accepts arbitrarily wide structured data blobs and we store the raw events. And in one point in the world, you did your aggregation at write time. You made your decisions about which questions you were going to be able to ask when you wrote the data out and you can't really ask new ones. With Honeycomb, we store these raw, wide events and then we do the aggregation at read time. So you can ask anything. You can be like, and not only that, but like you could ask the machine to do it for you. So one of my favorite things is called bubble up, which is just this thing, because we've stored all these raw events, any graph, any dashboard, any SLO that you have, if you see something, you're like, huh, I wonder what that is. You draw a little bubble around it. And we compute for all the dimensions inside the bubble versus the baseline outside the bubble and certain distance. You're like, what's this spike? And then immediately you can see, oh, inside the spike are their only events going from Android devices using this version from this link using this language pack using this build ID using this application ID using these feature flags, you know, because like so much of debugging is here's the thing I care about. Why? Why do I care about it? What's different about the same? And this is where I feel like the ways we've debugged in the past have relied on you knowing what the answer is before you can ask the question. It's very search first, right? You have to know what string to search for, what metrics to look for. You have to know what you're looking for before you can find it, which is just so ass backwards, but anyone can go I care about that Why you know and when it comes to debugging the debugging workflow should usually be either Here's my SLO. It's burning down faster than it should so tell me what's different about these events that are violating my SLO Versus the baseline. Oh now I immediately know what to go fix. That's like scenario one. And scenario two is, see, another thing about 1 .0 versus 2 .0 is that observability 1 .0 is very much about debugging, fixing problems, operating your code. And 2 .0 is like the substrate of observability driven development. It's like, it's what your development based on. It's what allows you to form these really tight feedback loops where you're instrumenting your code as you go, and you deploy it and it's there and you're like, okay, looking at the lens of the instrumentation I just wrote, what's different? Is it doing what I expected it to do? Does anything else look weird? You don't know if anything's wrong or not. But that point right there is your best chance to ever find a problem. And

that's kind of what you did in the Google Cloud Next keynote. You gave an example of this, and I think also on social media I've seen you post a few times, usually a retweet of someone who's like, hey, this happened. And it was observability driven development. Those are very interesting situations. Yeah,

because the cost of finding and fixing bugs goes up exponentially from the moment that they're written. If you don't find it shortly after you've deployed it, it's probably not going to be you who finds the problem. It's going to be a user or a customer or another engineer at some point down the line, months, years, never. And so finding that feedback loop and making it fast, and it's what makes engineering fun, is having this constant conversation with your code, just being able to move fast with so much confidence. The whole debate about should we deploy on Fridays or not, that speaks of so much fear to me, just this fear of, so often we deploy things and we have no idea if they work or not. We don't know what's going to page us. We're not confident that this worked. And of course they aren't confident because all they have are aggregates and like random exemplars. They can't actually see what they deployed and how it's different from when they before they deployed. Of course they're not confident. So I think it's really just like, it's like a sea change in how we think about building and shipping our code.

So it feels like this observability 2 .0 concept is really about being an engineer who is just fascinated with your code and you're excited to learn about how it works when it's out there in the field. And if you have the ability, the tooling where you can do something and then just like go and ask questions about it. I feel like everyone's asking those questions, but you usually can't see the answers.

Exactly, not only that, but you're used to getting punished for your curiosity. I remember like when we put a new engineer on call, and they start out curious, they're like, what is this? What is this? And you're just like, oh, grasshopper, don't pick up that rock. It's going to take you on a long and winding road. It's going to be five days later. You're not going to have gotten anything done. Just don't even look at it. You know? Well, that sucks. And you're going to end up in

all of these weird places and it's just going to be pain and suffering. Because it's

so true that like right now, everyone's distributed systems are broken in so many ways, like so many ways. One of the most entertaining things is like, we'll often do these POCs with prospects, right? And without fail, we'll be pairing with them, writing instrumentation or something. And our sales engineers start going, what's that? Are you about, is something wrong over here? Five minutes later, someone gets paged and they're like, how did you do that? You know, or, or sometimes they'll be like, they're like, ah, we have to stop and fix all these things. And our sales leaders would just be like, oh, no, we're never gonna get through the PSC if we don't just get through this. They've been broken for a long time, trust me. It's just that you never had the ability to see it. And it's so satisfying. It's so satisfying and it's so nice to feel confident about your work.

This is making you very curious about how databases factor into your book. And I need a book club to tell me go read that book. Because I actually have a meeting later today about databases. So I feel like this is going to come back into that. So, you know, I'll send you a couple links after this because someone who worked for

us briefly wrote two beautiful, like, my mom and dad would understand these pieces, I think. Just essays about database internals and how they roll up to observability. And they're just such fun reads. You'll love them. I'm

very interested. So we'll make sure to include in the show notes. I look forward to it. And so I think we should start wrapping things up. This is so fascinating and I feel like it's just opening a door. Observability I'm realizing is what I want from so many of the systems in my life. Yes. And so I just want to go learn about observability. This 2 .0 concept because it's just about being fascinated and observing. your systems, learning from your systems. It's a reasonable request. Everyone wants to understand their lives. Yes. Very interesting. I'm excited to dive into that. But one last thing that I want to ask you about, since I am such a big fan of your blog, I have noticed that any time I attend a coffee ops event, which for anyone listening who is not familiar with coffee ops, it's this like meet up series. It's like a global concept where folks get together and you sit in roundtables usually and you have post -it notes and everybody writes ideas of things they want to talk about on post -it notes. And then everybody votes. You have a certain number of votes. And so you talk about the top topics for like five minutes and then folks decide if they want to move on to the next topic or not. Anyway, every time I go to one of these, I bring up blogging and everybody wants to talk about it. As someone who's blogging, I really admire. Do you have any advice for folks out there who want to write blogs? Sure.

You know, I have a goal for myself of writing one post a month and I've never actually achieved it. Can relate. But you know, goals are important. So one easy tip is if you ever write a talk, write a blog post out of it too. The reach of blogs is actually farther than even the best talks. They're indexed, people, you know, link them. And if you've already done the work to like pull together the material, it's so like, and anytime you write a blog post, it's good. Honestly, you can convert it into a talk. It's just really easy Number two, I think people get kind of all up in their own heads about is this original or not? Has anyone ever talked about this? Is anyone going to be interested in this if you're interested in it? Write about it, you know because it's the interest in the passion that comes across nobody really wants, you know, nobody really cares Oh has this been said before? The best messages are the ones that get said over and over and over because they affect lots and lots of people So it doesn't matter by chiming in you're saying yes, this matters to me too And that means you're contributing to the growth and the distribution of things that you care about And don't feel like you have to be the voice of authority. You know, you're the voice of the authority of your experience. And if you've had an experience that is sticking with you, it will probably resonate. It doesn't have to resonate with everyone. It can resonate with other people like you. You know, some of my favorite talks and writings are about subjects that I know very well. It's just really fun to see a new fresh take. It's fun to see someone experience it for the first time. You know, these are perennial topics. Like people are always learning these things. And so think less about what you should do and think about what does interest you and what does stick with you and write about those things. Also, shorter is better than longer. Advice that I never take.

I will admit that your posts are long, but every time. And I have a lot of trouble reading long blog posts, but I never have trouble reading yours because they're just so interesting. The passion really comes across. And I think that's a throughout our whole conversation today is kind of that fascination and passion. Whether we're talking about engineering management and doing your job as an engineer, you need to have that passion and that enthusiasm for it. Observability is about feeding that that passion and that fascination with the work that you're doing and blogging is about sharing that fascination and passion with the world. Yeah. I

love that. Thank you, Kaselyn. That's

always my advice for folks who want to give talks or write blogs or anything, honestly. The best content that you create is going to be about something that you just really care about. Yeah. It doesn't matter what it is. It doesn't matter if it's been done a million times. If you care about it, then it'll be interesting. I agreed. Wonderful. Thank you so much, Charity. As I said at the beginning, it really is an honor to talk to you because I just love your work. I love what you're doing. I can't wait to see more of it.

Thank you so much for having me. This is really fun.

Yeah, thank you for being on. Well,

thank you, for that interview.

Welcome back, Avdel. You've been just all over the place lately. Yes. Yes,

so many things happening in Europe. It's crazy. You

even gave a keynote, right?