The One Where We Geek Out on Observability with Charity Majors

7 snips

Jan 16, 2024

Ask episode

Chapters

Transcript

Episode notes

About our guest:

Charity is an ops engineer and accidental startup founder at honeycomb.io. Before this she worked at Parse, Facebook, and Linden Lab on infrastructure and developer tools, and always seemed to wind up running the databases. She is the co-author of O'Reilly's Database Reliability Engineering and Observability Engineering, and loves free speech, free software, and single malt scotch.

Find our guest on:

Find us on:

All of our social channels are on bento.me/geekingout
All of Adriana's social channels are on bento.me/adrianamvillela

Show Links:

Additional Links:

Transcript:

ADRIANA: Hey, y'all, welcome to Geeking Out. The podcast about all geeky aspects of software delivery. DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And geeking out with me today...I am so excited to have Charity Majors of Honeycomb on! Welcome, Charity.

CHARITY: Yay! Thank you for having me, Adriana.

ADRIANA: I'm so excited. And where are you calling in from today, Charity?

CHARITY: San Francisco. I just got home. I was in Charlottesville, Virginia, with my little sister over Christmas, and so I am newly home again, looking forward to a very quiet week between Christmas and New Year's.

ADRIANA: That is always the best week for chillaxing, right?

CHARITY: Nothing going on. This is why at honeycomb, we just give everyone the week off. Obviously, some people have to be on call, but why pretend you're getting stuff done if you aren't?

ADRIANA: I know, right? Yeah, I fully support that. I totally agree. I think more companies should embrace that.

CHARITY: Yeah. I don't feel like anyone should have to be performing that they're excited to be at work or like, we don't make people have a set number of vacation days or anything, but...That's the worst. If you're like, well, it wouldn't really be working, but do I spend one of my precious vacation days? Yeah, fuck it.

ADRIANA: Yeah, I agree. Honestly, I get so much anxiety over vacation days, like, having to meticulously plan them and, like, oh, where do I spend them? And maximize vacation with family and school holidays. And there's, like, so many school holidays, right?

CHARITY: Seriously, there's no perfect system. Like, if you do the unlimited holiday thing, people are like, well, but then you're not treating it like real comp. And people have stress about, are they hitting the right number of days or not? And people won't take it. But then if you have specific number of vacation days, then it's where do I spend it? And everything. So I guess if there's one thing that being a CEO CTO of a company has taught me, it's that people are going to complain no matter what. All you can try and do is pick what is genuinely best for your people that will really help you get as much work as possible done without asking people to fake it and do a bunch of. So, we've gone the infinite vacation route, because, all things considered, I think you kind of want to have a mandatory minimum. Like, you have to take two weeks off, right?

ADRIANA: Yeah.

CHARITY: And above and beyond that, it's like, are you getting your work done here? It's a standard. The company standard is about three weeks a year, but nobody's looking over your shoulder and policing you.

ADRIANA: Yeah. See, I appreciate those policies, especially at companies where they fully respect autonomy, because there's the companies where it's like, well, it's unlimited, but we really only expect you to take like three weeks or four weeks or whatever, and it's like, so it's not really unlimited. Right. And that's disingenuous and annoying and very stressful. I don't know. I bust my ass and I need the time to chill.

CHARITY: Yeah. But I will say some people will start taking five weeks, six weeks. But then the question that you have to ask them is, you're taking too much time. It's like, well, are you really getting your job done? And what's the impact on the people around you? Really?

ADRIANA: Yes.

CHARITY: Because, yeah, it isn't actually fair if you take eight weeks off. Anyone would understand if you have a health issue or if someone in your family is. We've had those situations. But if you're working at a startup with some intensity, we have VC money that's burning in the bank. You kind of can't get your job done, really, if you're not there for two months out of the year.

ADRIANA: Oh, yeah.

CHARITY: I think always trying to steer it back to the impact. Right. Can you get your job done and are you letting down the people around you, or are you being a real functional member of a high performing team? Those are the terms to have this debate on not how many days you're here or not. The other thing, unlimited time, is that it removes the aspect of scorekeeping and time keeping and quibbling about hours, because some people don't really care, but some people get really concerned about, well, am I taking 2 hours off here and 3 hours there? If I take 4 hours of that a day or not? And those are brain cells that I would really rather you just devote to solving the problems that we're paying you to solve, not to bookkeeping around your own anxiety or your projected expectation of someone else's anxiety about the hours that you're spending on your job.

ADRIANA: Yeah, absolutely. I have to admit, the timekeeping stuff is so stressful, and I've been lucky the last three years. I have not had to fill out any timesheets, which has been like, oh, my God, my first job out of college was, like, consulting. So all of your fucking hours are accounted for.

CHARITY: Oh...

ADRIANA: So everything and even your downtime, right? If you're in between projects, you got to charge it to internal thing. And it was like, yeah, I lasted four years.

CHARITY: Oh, honey. I don't know how! One of our company values is we hire adults. And I actually think about that. It's as much about us as it is about the people we hire. It's like, are we treating people like adults? Do we expect them to manage their own time or not? And of course, the difficult points come. I think as an industry, we're just terrible at figuring out how to really take people on as apprentices and turn them into fully-fledged employees. I mean, there's that middle section that takes, even for a fresh college grad or someone entering...It takes five to seven years, I think, for you, really, to bring someone on and bring them up to a level of senior engineer and teach them all these things.

But you can interpret it, our value as you're on your own. You better come fully baked because we're not going to help you, which is not what we're trying to project or do. But it's challenging, no?

ADRIANA: Yeah. It's so challenging, like coming out of school, right? Trying to figure out where you fit in. And it's also kind of, for me, it was like a bit of a mind fuck because I was like the goody goody. Like, I will do all the assignments. And marks were everything. And then you go out into the real world and it's like, yeah, bye bye. That did not apply. For me, it was a massive adjustment and I kind of sucked fresh out of school, like my first couple of years in the work world trying to figure out, what do I do? What do I do? There's like, no marks. Not in the standard sense, right?

CHARITY: No, of course not. You must be an upholder type. Do you get a lot of satisfaction out of checklists? Like your own checklists and the checklists that people do?

ADRIANA: I do, I do. My own checklist. My whiteboard next to me. It's mostly clean now because of the holidays, but it had my to-dos...but I've had to learn to roll with it. I had to be a lot less uptight than I was in school, because I think you just have to, in the work world.

CHARITY: Well, because you learn eventually that if you want to be successful, it's not actually about checklists, it's about figuring out what matters to you and what matters to other people and then figuring out how to creatively achieve those goals. And the checklists are there as a tool, right? I'm not telling you anything you don't know.

ADRIANA: Yeah, exactly. Yeah. I completely agree. And I think that's a lesson that comes so much more easily for some than others for sure. Especially. I've hired a couple of interns in my past life and trying to steer them in the direction of, like, chill. Let's relax. Let's just focus on getting the work done and learning cool shit.

CHARITY: In a lot of ways, though, I would argue that the upholder is the easiest type of person to onboard because they're motivated by everything.

ADRIANA: True.

CHARITY: So when I use the term upholder, I don't know if you've read the book, "The Four Tendencies"? It's this book that it's super cheesy and I don't want to get anybody's expectations up, but it was actually really pivotal for me and Christine [CEO of Honeycomb] and finding a way through our relationship because she's an upholder. I'm the opposite. I'm a rebel. Which means that I reject all of your checklists and my own too, called checklist. Basically, it's about motivation. And there's only four possible types.

It's a two by two, right? It's like your own motivation, like what motivates you and the goals that you set for yourself and then the goals that other people have for you. And you can either be super motivated by both or you can be what's called a questioner type, which you can't really give a fuck about other people's expectations. But if you care about something, then you can hit that goal every time. And then there's the type that needs a gym buddy because you struggle to do the things that you set for yourself, but you respond really well to external structure. And then there's the type that rejects all of the structures. And that's my type. And this was really helpful to us in just like, sort of because Christine and me are just such polar opposites that she was just like, who the fuck are you? How does your brain work? Why is it that I give you this perfectly formed challenge and you're like, "Fuck all your challenges." And I'm just like, "Why are you telling me what the fuck to do? Don't you know that's the easiest way to demotivate me, is to tell me what to do?"

And so it was really helpful because this book actually has these almost, like, examples of, if you're this type in a relationship with this type, here are some conflicts and conversations that you might have if you're in a working relationship and you're this type paired with this type. And it was just like, oh, my God. Some conversations that I had had with my partner, like almost word for word, some conflicts Christine and I had had, almost word for word. It was just like, here are some tools for getting around them. So I really like it.

ADRIANA: That is so helpful. It's funny, because I think the way you describe yourself is how I would describe my daughter, too, to a certain extent, because when she was in preschool, her teacher could not teach her, and she realized that the way to teach her was not to teach her, but to teach her friends. And then it would cause Hannah to go over, oh, that looks interesting. So she's like, don't tell me what the fuck to do. I'm from Brazil. And I'm like, oh, it'd be so cool if you learned Portuguese. She's like, "No." What did she do? She learned German.

CHARITY: That is how you deal with rebels. You have to rely on them to find their own intrinsic motivation, because if it becomes part of their identity and part of who they say that they are, then you can't stop them.

ADRIANA: Yeah, exactly. Yeah. So I'm like, you know what? You do you. I embrace that. And I think she's happier for it. I'm happier for it.

CHARITY: Everyone should be happier for it. As a manager, part of what you have to do is, I feel like, as a manager, in the beginning, we try to give our reports the experience that we wish we had had. For upholders and for...I can never remember...the obligers. Obligers are the ones that need the external structure. You're really giving them a gift. If you give them a structure or if you give them regular check ins and you let them know what the expectations are, you're giving them a huge gift, and they will rise to the occasion and they'll thank you for it. And if you do that for rebels or questioners, you're insulting them.

That sort of versatility. And it's not just managers, of course. It's anyone who's, like, in a senior plus position, where what you need to do depends a lot in influencing others. Just sort of having a mental map of how other people respond to sort of motivations is super helpful.

ADRIANA: Yeah. I actually remember reading one of your blog posts on, like, I think you're talking, like, being manager and trying to make everybody happy, but it's not also about being their buddy and making everybody happy, but also, you do have company goals to fulfill. And so to what extent do you protect your team, but then don't end up doing the things that need to be done, which I think is such a common pitfall for new managers, because for me, certainly when I first got into a management role. I'm like, this happened to me.

ADRIANA: I'm not going to let that happen to my direct reports. I am going to be the best manager that I can possibly be. Right. It can kind of blow up in your face if you're not careful. Like, I wanted to be friends with my direct reports. That did not work out in the long run. Initially, it was like, yaaaay. But afterwards, it was like, no.

CHARITY: We're always overcompensating for our own experience.

ADRIANA: Yeah, exactly. And in the end, I think we learn, right?

CHARITY: Yeah, exactly. Eventually, hopefully, we find a happy medium. I think about that so often when thinking about diversity issues in the industry or about management or that it's natural for there to be like, this is a young industry. This is a very young profession. For as old as some of us feel like we are, we're still like, there's been... When I was coming up, we didn't talk about women in tech. There was a few of us that were just, like, quietly there, wearing men's clothes and just sort of pretending we were straight white dudes. And so there was a lash, right? And then there was a backlash.

And it swings. I'm not going to say too much about how sensitive I think some people are, but I understand why they are. I understand why they are. And also, that's not where we have to end. That can't be where we end up. We have to end up in a place that is less reactionary on all sides.

ADRIANA: Absolutely.

CHARITY: The goal of our businesses and our companies, this is something I've been thinking about a lot. The few times that I feel like the honeycomb culture has gone off the rails a little bit, is when we've kind of lost sight of the fact that we are here to serve our customers. We are not here to have the most diverse company in the world. We're not here to give people the best work life balance. We aren't even here to give everyone the best employment experience of their lives, which early in our, when it seems for so many years like we were going to fail, Christine and I would console each other. We'd be, you know, if we go under tomorrow, as we think we probably will, at least I think we've done a good job of giving a lot of people an experience that will set the know so they won't accept shitty jobs for the rest of their life. But now that we're hoping to be around for a long time, we can't forget that we are here to serve our customers. The decisions that we happen to think that a lot of these things go in harmony.

Treating people really well means we treat our customers well. Having people who are happy at work. We believe in having healthy businesses, which is a lot of people's complaints. They see symptoms, but what they'reacting to is the fact that the business is not healthy. The way people are relating to each other is not healthy. I wrote this other blog post a while ago, I don't know if you saw it about, "Choose Boring Culture"?

ADRIANA: That sounds vaguely familiar.

CHARITY: You know, because Dan McKinley wrote that blog post that was hugely influential on me about choose boring technology where he's like, you know, as a startup you get three innovation tokens. Choose wisely. And I feel know the same is true for culture and businesses. And like, we stand on the shoulders of...you know, a lot of people, a lot of really smart people have figured out things about how to make companies work well. There's this great book by Pat Lancioni called the Advantage, which I think of as like the James Madison of business and organizational structure. He's incredibly innovative thinker and he makes things very simple. But he's like, the advantage increasingly in corporations is not your widgets. Because everybody's widgets are getting so good. It's how healthy is your organization, which means how much of your people's creativity are you really taking advantage of? How much of their creativity do you feel free to bring to work? Is your organization equipped to absorb it and to change from it and to react to it? Are you able to keep people who are passionate about their work? Do you let people go who are detracting from the culture? And he's like, it is amazing how poorly most organizations are run to this day.

So choose boring culture. I think in a lot of ways, companies don't have to make their companies interesting and fun because people will do that. People have so much fun, creative energy in themselves. You just have to create a boring place for them to work where they can do their best work and they'll come up with all the fun stuff.

ADRIANA: Oh, I love that. That's so cool. You touched upon something that I am a huge proponent of, which is like, letting go of people who are not adding to your corporate culture. Because I think there's this tendency, I think, in our industry to hire rock stars and kind of ignore the shittiness and their personality because, oh my God, they're the best of the best at blah. Right? And I've personally experienced a couple of incidents in my life where if you have somebody who is constantly just being negative on your team, no matter how good the rest of your team is if they're like, poo pooing everything, it sullies the culture. It's like a poison pill. And it's not like, oh, I'm going to fire your ass. It's like, well, perhaps this team might not be the best for what you want to achieve. Perhaps I can help you find a position in another team in the company. Because it's just poison.

CHARITY: I think it starts with not having kid gloves on. I don't think you jump straight to firing. I don't even think you jump straight to moving. A lot of these people have never really been told no in their lives. And some of them can take it, some of them can. But I think you owe it to them to figure it out, right? To start giving feedback consistently and regularly working with the person. And this is something that I think can be really frustrating to people who are. When it looks like management is doing nothing right, because it looks like, I know that people at Honeycomb have felt this way at times, because it looks like they're just kind of being shitty and they get better and then they don't.

And it's always a judgment call. And I would actually agree that we always probably wait a little too long in general, but we waited a little too long with everyone. And I would take that over being a little too fast to fire people, because I think that that even more trust. But, yeah, I agree. If they can't bend, if they can't change, if they can't understand that the smallest unit of software ownership is the team, it's not the person. It doesn't matter how great one person is, because one person can't own software. It's all about, are you contributing to the overall greatness of this team? You can bend your rockstar talents to that, but if you're not willing to, or if you can't, then there's no place here for you. I'm sure you can get paid a lot more money somewhere else.

ADRIANA: Yeah, absolutely true. Absolutely.

CHARITY: Sorry, go ahead. I didn't mean to cut you off.

ADRIANA: Oh, no, I was just saying I agree with you, but I think that.

CHARITY: Letting go of people is hard, and I think that it comes in all forms. I think that it's really discouraging to people who are on a high performance, who want to be on a high performing team, when someone isn't really showing up and who consistently isn't showing. The person who's like, consistently taking six weeks of vacation when everyone else is taking three or four, or the person who is kind of half asking it. And all of us half ass it sometimes, right? But people can tell you work on a team for a while, you get a real good sense of how hard everyone is working, how much they're trying. Sometimes it comes in form of, this is almost some of the most heartbreaking ones of when you've got someone who's very junior who just isn't working hard enough. And it's like we kind of don't have the language to tell them that. Because on this pendulum, we're so far over to the side of, you shouldn't be like, work crush code. It's almost like we've kind of lost the ability to tell people, no, really, you're probably not going to make it if you don't put in a few more hours and if you don't have a little bit more grit.

And some people don't want to work that hard, and that's fine, but you aren't automatically granted a job based on however hard you do or don't want to work.

ADRIANA: Yeah. And it's such a tough conversation to have. I had someone in a previous team that I hired on as a senior person, and then she was, like, scamming on my. She was scamming on everyone else. She would just pretend that she was doing work by, like, oh, let me attend meetings with so and so. And meanwhile, I'd hired this junior person who was working like she was working at the senior level. And it was so frustrating. I was trying to have the conversations with the senior person saying, listen, I want to help you. How can we work together? But she got offended. And these conversations are so hard to have because we all perceive differently how we're doing. And in her mind, she was doing just fine. How have you had those conversations in the past with people?

CHARITY: Oh, it's really hard. There's no version of this that isn't hard if you care about people.

ADRIANA: Yeah.

CHARITY: My most recent blog post was about why anyone should go into engineering management. Because it's a hard fucking job. And the answer is, because we need them. Because we need them desperately. Like a team with a great engineering manager builds circles around teams without one. And the other reason in my piece, I said is that it changes you as a person, and it gives you these skills that a lot of us didn't learn when we were growing up about how to be honest and how to have hard conversations and all these things. But as to your question, how do you go into this? The number one thing I think is no review should be a surprise. You should be having this conversation consistently, which is a hard thing to do because it makes people feel demotivated and frustrated.

But sometimes they have to feel that way. We've instituted a rule at honeycomb that if you're thinking of putting someone on a PIP, if you're thinking of, you have to literally say the words, your job is at risk because it's so tempting when you're face to face with someone who you really want to succeed, to soft pedal it or for them to feel upset and for you to kind of walk it back, or for you just to use words that let them walk away thinking something that is not what you want. And there are tools you can use to make sure. You can write up an email afterwards to be like, just to be clear, this is what I saw. This is what I'm saying. This is what you're hearing. But I really do think that one of the most important tools we have is just being explicit because they can file it away. We all have such infinite creativity when it comes to explaining away things that we don't want to hear.

And we can be like, oh, my manager is kind of a bitch. Oh, they're just in a bad mood. Oh, they're just kind of riding me lately. Oh, it's because of this thing. But this will be over. And I feel like if something really isn't trending, well, we have a responsibility to be more of a dick. We have to be the ones who kind of put our bodies in the breach and be like...and just sit there and deal with their reactions, which are going...They're going to have negative feelings. And it's really hard to sit with someone else's negative feelings who you are the proximate cause of. It's really hard, but you have to do it. It is the best thing for them to do it, to let them know this isn't just a small thing. This isn't just a flash in the pan. You are not succeeding. You are not on a path to succeeding here. You are on a path to, your job is at risk. Honestly, that's the kindest thing you can do for someone.

ADRIANA: Yeah, that makes so much sense. And you're right. It's so hard to get those words out. Like, "Your job is at risk." Yeah. And I've worked in organizations, too, where pussyfooting around the topic was like kind of the cultural norm, and so things wouldn't get said that should have been said, and you don't have the favorable outcomes in the end.

CHARITY: Yeah. And then people feel stabbed in the back, understandably. I would, too. They go...walk away going, "If they had just told me, if I had only known." And that is the worst outcome. That is the thing that I always remind myself of when I'm just like, I love this person. I don't want to be mean to them, but I cannot take it if they walk away feeling like I didn't tell them, like I stabbed them in the back by not making it perfectly clear that they're not performing and their job is at risk.

ADRIANA: Yeah, it's definitely something that I wish that I had done more of in the past, and I try to remind myself of it, but, yeah, I think that is absolutely the right thing.

CHARITY: And to your point earlier about being people's friends, you can absolutely be friends with your direct report, but there's a line there. There's a boundary there, and there's a point at which you're not their friend. It's just like being someone's parent, right? When things are going great, yeah, you act like friends, but they have to know that when it's time for you to be parent, you're going to be parent.

ADRIANA: Yeah, exactly. Because otherwise they will take advantage of you.

CHARITY: Right. They will completely take advantage of you. It's human nature.

ADRIANA: Exactly. And you will let your guard down, too, right? Because they're like, oh, "I don't want to hurt so and so's feelings, otherwise they won't love me." And it's like, you kind of have to get over that as a manager. And it's hard.

CHARITY: It's really hard. It's really hard. And it's always a matter of judgment. It's always a judgment call. And you have to know that after you've had that hard conversation, chances are they're going to go tell other teammates a version of it that makes you look bad and them look great. And you can't do fuck all about it. You have to sit there and take it and hope that the relationships and the trust that you have built up are enough that people aren't going to just automatically believe that other person. That is the hardest thing about being a manager to me.

CHARITY: That right there, knowing...is when I know I can't say anything.

ADRIANA: Yeah. And risking, as you said, having people say, well, management doesn't know what they're doing. Oh, my God. Because as an IC in the past, I was like, management clearly doesn't know what they're doing, and then...

CHARITY: Clearly doesn't know shit.

ADRIANA: The first time it happened to me, oh, my God, I want to go cry. Like I'm trying everything to make you happy.

CHARITY: Yeah. This is why I feel like my dream vision for the future of engineering management is that more people do it. But people don't do it. They don't do it as a career. They do it as a tour of duty, because I feel like having ex managers on the team, it's like a game-changer, because whenever the dynamic is ICS versus managers, which always happens. Comes and goes, but it always happens. It's so helpful to have an ex-manager there on the IC side who could go, okay, kids, it might be this. It might be this. It might be this. Do we trust this manager in general? Okay, well, let's not jump to the automatic conclusion that they're just an idiot or they're just, like, being manipulated by the upper or whatever. They're the only voice in the room who can talk people down off a cliff and remind them whether to have some trust. And it's such a game changer. It is so wonderful.

ADRIANA: Yeah, that is so true. And it makes so much sense. I even find myself in positions after I've been a manager, and then being now an IC...whenever I get comments...

CHARITY: It's nice!

ADRIANA: Yeah, it is nice! And sometimes I have my manager apologize, "Oh, I'm so sorry. Blah, blah, blah." I'm like, "Dude, I totally get it." "It's fine. No worries."

CHARITY: You're able to give so much better support and understanding to your manager than you ever could have without that experience.

ADRIANA: Exactly.

CHARITY: It's so grounding and validating for them to have someone who sees them.

ADRIANA: Yeah. And especially, also when you have that nice rapport with your manager where you have that ultimate trust, where, okay, it might seem like they're riding you hard, but then you're like, oh, my ex-manager brain has said, okay, "I have a good reason to trust them. Take a step back. Let's look at the big picture." And, yeah, it's cathartic and it's eye opening.

CHARITY: Everyone wins.

ADRIANA: Yeah, exactly. No, sorry. Go ahead. No, please.

CHARITY: I often hear people who are first-time managers who are, like, anxious or like, if I go back to being an IC, will I ever get the chance again to be a manager? And I'm just like, "Oh, grasshopper, they can smell it on you. You will be fighting off manager opportunities for the rest of your career." Have you found this to be true? I expect you have.

ADRIANA: Yeah, I have. And it was funny because after I read it in one of your blog posts, I was like, oh, yeah, so true.

CHARITY: Yeah.

ADRIANA: I mean, it's on your resume. Yeah.

CHARITY: Just the way you come across. I've also said that the fastest way to mint like, a shiny new staff engineer is to take a senior engineer and put them in management for a couple of years. Because the way you present yourself at work, the way you approach problems, you have such a better sense of the business, even if it wasn't on your resume. This is why some people get to be managers early and often, because for whatever reason, they already have some of those skills. But once you've been a manager, it's written all over your face that you understand.

ADRIANA: Yeah, very true. Now, here's a question for you. What's your take on folks who have gone into management at a really early point in their career, becoming a technical manager for a technical team when they don't have that many years of actual technical experience?

CHARITY: I think they are not well-served by this. I often see this happen to women, especially, and I think it's often intended as a compliment and by people who genuinely are trying to do they want to help the industry. They know that there needs to be more women in leadership and management. And so they're like, here's this person who has social skills and also some engineering skills. So we'll just...I think everyone has the best of intentions, and I think it really does not serve them because it's often a one-way...it's a one way-ticket, right? Because you don't have the skills to be able to go back and pick up coding easily in a couple of years. I think you also don't really have the skills to be a great manager.

Honestly, my recommendation to them would be get back to coding as quickly as you can or climb the ladder. If you choose to climb the ladder, then those skills are less relevant. But I wouldn't be in a rush. If you're 25 and you're a manager getting offered a director position, I would look at that cross-eyed. I would be like, because, yes, it is probably a compliment, but is it the right thing for you? I don't know. I mean, if you play out over the course of your career, you've got a 30, 40 year career. There's no rush. And the people who really excel in those senior leadership positions tend to be ones with deep roots, not just a very shallow.

And there's so much to learn, right? This is not to say that there's not anyone out there who's climbed the ladder in a hurry and not regretted it, because there probably is. But the people that I know who have done it have, by and large, profoundly regretted it. You know, I wrote about my friend Molly, who's an engineer at Honeycomb now, and she was one of those people. She super bright, straight out of college, became an engineer, became a manager, became a director. Shot up. You know she was a VP, she was a director, she was an EP. And she came to Honeycomb to be our head of...VP of customer success or something like that. And she was so unhappy.

And she would make all these wistful comments about how she wished she could be a software engineer. She wished she had done that. Eventually, her husband, he was an early member at Okta and Okta IPOed. And so suddenly she was like, "Wow, I can do anything I want with my life. I want to be a software engineer." And so she became a support engineer for us, and she just started writing code on the side. She started picking up some PRs. Now she's a software engineer on the team, and it's been hard.

She's never been happier, though. And I'm proud that Honeycomb is the kind of place that can support someone in doing that, because I think the opportunities to do something like that are few and far between. There are not many places we'll take a flyer on someone who's middle-aged and wants to go back to software engineering. But if you think of your career as a long game, you don't want to amass a bunch of titles, especially titles that are kind of empty because you're not getting a...I would...I would venture to guess that you're not getting a really high quality offer to be a director or a VP at age 27. It's really mostly the title. You want to amass yourself a solid base of experiences and skills, and you want to have shit to draw on as you climb that ladder so that you can help people better.

So the thing that I do want to guard against when I'm talking about this, I'm speaking to people who are early in their career, who are facing these questions. I don't want to make it sound like it's too late and you're screwed if you're already in this position. In fact, if you're in that position, if you'd like someone to talk it through, reach out to me. I have a Calendly link, calendly.com/charitym/advice, and I'm always happy to talk through interesting and tough career conversations with people. You have skills, you have assets. It might not be a super sexy path, but you can find places that will take advantage of the skills you have to offer while you kind of work your way up from the bottom again, if that's what you want to do. I'm sure you can do it, but it's easier if you do it right the first way and become a solidly senior engineer. Seven years really is the minimum, I think, before you become a manager.

And if you really want to be able to manage other senior engineers, you need to at least be able to speak the language and be able to roll back on it.

ADRIANA: Yeah, I fully agree with you on that. I was thinking back to my own career. My first job out of school was as a consultant at Accenture, and the career path was basically like, you must pay your dues as a developer, and you shall be rewarded with a management position. Right? Yeah. Right. So we're all kind of brainwashed to think, oh, my God, if I'm not a manager, by 27, 28, I have failed at life. Right? And I hit this crossroads in my life where I was being groomed to be a manager. I didn't have the manager title, but they threw me on some engagement where I was managing three teams at once. I was doing a shitty job, and I'm like, I was miserable, and I'm like, what do I want to do with my life? And so I decided...I left consulting. I took on a job as a software engineer. It was a lateral move, but I was so happy, and it was the best thing for me because my thought was, how can I manage these people if I don't know enough? I just didn't feel right for me, so I'm happy I did that.

CHARITY: Good for you for listening to your gut. I think all too often we talk about impostor syndrome, and we try to talk people out of it. I often think if your gut is really eating at you, that something is wrong. You should listen to that. You shouldn't just go, oh, everybody, there's impostor syndrome, and then there's just, like, the feeling in your stomach that you're not really setting your future self up for success or that you aren't really equipped to do the kind of job that you want to be able to do in this role. And I think that is not something to be brushed aside lightly.

ADRIANA: Yeah, I definitely agree. Listen to your gut, because it's telling you something. One thing that I wanted to ask was, when you were building Honeycomb from the ground up, did you have sort of lofty aspirations of how you wanted things to be?

CHARITY: Ha!

ADRIANA: How was...the initial thoughts versus how it turned out?

CHARITY: I 100,000% expected us to fail like the plan was to fail. So I was never one of those kids who was like, I'm going to start a company. Because I always kind of low key hate those people. It's like, "Oh, you're too good to work for someone else." I'm not too good to work for someone else. I was a serial dropout. I'm the opposite of you, right? I didn't collect all the awards. I didn't check anything off.

I dropped out and I dropped out again. I dropped out again. And so I never had a pedigree. Nobody was ever going to give me money. Then I was leaving Facebook, and the first time in my life, I kind of had a pedigree. And so I was like, well, I can't waste, like, on behalf of all women and queers and dropouts everywhere, I have to take it and run with it and do something. But I was super burned out. And I was like, well, I guess I have an idea, but I'll go heads down the corner, write code for a couple of years, and then we'll fail.

And I'll open source it. Then I'll have my tool to use. Hee haw! That was really the grand vision. And I would say Honeycomb has been around for eight years as of January 1, but we had many near-death experiences. Now, we hit our $40 million ARR mark this month, which is exciting. We're hoping to get on a path towards an IPO. But for the first five years, I think we wobbled around between 5 people, 12 people, 30 people. We did layoffs down to 15 people again. We were a skeleton crew wandering in the wilderness. In retrospect, I realized that we were creating a category and we were writing the database and all this stuff, but it just felt brutal. It just felt like failure was around every corner. And most of those corners were right. We did fail most of those corners. There are several just, like, near-death experiences that we had, and we made it through.

And now I, for the first time, am not thinking we're going to fail. But no, there was no grand vision. There was no grand vision at all. There was just, like putting 1 foot in front of the other and feeling like I was failing the people that I loved most almost every single day. It was brutal. I will say, though, that Christine and I, a little bit older than your average tech founders, especially me, and turns out we have very strong opinions. And we learned a lot of lessons at previous startups. We were at, like, at Parse, which I loved working at Parse.

Parse is where I learned about the importance of design, about marketing. People loved that product and I loved working on it. Before Parse, I was like, I'm just a backend engineer. I don't care what the product's about. I'll work on anything. Parse is where I learned that, of course, that was never true. But Parse never really had a shot because the founders never really tried to build a business. They tried to build a great product, and they did. But then around series B, they had a marketing person and a couple salespeople.

We weren't bringing any revenue. They had to sell. Their destiny got taken out of their hands because they had no other choice. And so Christine and I, from the very beginning were like, we want to build a business. We want to build a business. We want to build a product that people want to pay money for. We're not building freebies. We're going to try and monetize on the other end of the pipe.

We are building a product. We're building a business. And I had a lot of just, like, very strong opinions about the kind of culture we wanted to build, just about how...in the beginning, when we were interviewing engineers, if anyone talked, not even dismissingly, about go to market functions like sales or marketing, even just sort of, like, almost alienated, just like, "Oh, well, that's them. We're us. We don't understand that." Those weren't our engineers, because we don't need to hire engineers who wanted to build a business with us and who weren't going to create that us versus them dynamic that makes all great business people in the valley feel like second-class citizens. So, yeah, I would say we discovered the grand vision along the way. It really wasn't there from the beginning.

ADRIANA: And as a follow-up, you know, one of the things that I admire so much about Honeycomb is you build such a lovely community around your product. Your customers truly, truly love it. And we met because I was asking so many questions in the Honeycomb Pollinators Slack. At the time, I was exploring Honeycomb as a potential product that the company I was working for might switch over to. And everyone was just so genuinely nice in helping me understand this Observability thing that was so nebulous. How do you build that thoughtful community? Was it something that you sought out to do from the get-go? Is it something that organically grew?

CHARITY: If you ask any founder, they'd say they're trying to build that, right? So I think the questions were like, "Why were we more successful than many others?" I think a lot of it has to do with just...and if you had asked me if I would be talking about values and shit, like a year ago or a few years ago, I'd be, like, rolling my eyes, because I've always hated when people are like, "Values," because most businesses are just like. I don't know. I get really cynical about it, but I feel like we are our customers, and our customers are us. We built this product to solve a real problem that we are having. And it is more important to us that these problems get solved than that Honeycomb is successful. I think I can say that about everyone there.

We would love to be successful. We'd love to make lots of money and all this stuff. But we see the pain that so many teams are in, and we know that we have a way to fix a lot of that pain, because we've seen our customers do this over and over, and we hear what they say about how no one else could do this. And we had the advantage of designing and building this 25 years after metrics began dominating the landscape. So we build on the shoulders of giants, like I said earlier. So I feel like it's easy to be a true believer, because we're not just trying to sell something. We're really building something that really changes people's lives. And it's easy to get starry-eyed about that.

It's easy to be a believer when you're all on the same page about fixing problems, not just about trying to tweak your messaging or your marketing or your sales or something. I think people, Honeycomb, are generally very passionate about solving the problem, and it's very exciting to see them. I mean, the product does what it says on the sticker, which is very exciting, because almost no products do. Most products are hyped. If anything, Honeycomb is underhyped. It does so much more than we've been able to explain to people, which is why our churn is like nothing. We win, like, 80% of our tech evals, which the industry standard is, like 30 or 35%. Once people see it on their data, you cannot pry it out of their cold, dead hands.

One of our best sources of leads is when engineers change jobs and they bring us with them, because once they've tried developing with Honeycomb, they can't go back to not having honeycomb. And this is all stuff that it's hard to explain to people in words, but once they see it, it clicks. And so, really, our core challenge, over the next year, we've built the product. Our core challenge is figure out how to get more people to click with it faster, because we know that once they've seen it. The deal is done, but it's still a very hard problem.

ADRIANA: Yeah. The other thing that I think is very interesting about Honeycomb is it's not only are you building a product that people are excited about, but you've also really turned the whole area of Observability on its head. I'd like to think that it was Honeycomb that sort of gave Observability...Observability became what it is because of what Honeycomb has done. I mean, you've spent a lot of time talking about Observability. I mean, honestly, that's how I got dialed into what Observability was in the first place, was catching your Tweets. Yeah, if you could say a little bit more about that.

CHARITY: Yeah. Like, Christine and I are not marketing people. It turns out what we were doing was category creation. All I knew was that we were trying to build something based on an experience we had had that had changed us as engineers, and we knew that it wasn't monitoring. And I spent months just sort of, like, testing language, trying various things. And one point, it was July in 2016 that I Googled the term "Observability", and I read the control theory definition, and I was like, "Oh, shit. This is what we're trying to do. We're trying to build something to let engineers understand the inner workings of a system, no matter what's happening, just by observing its outputs."

So, like, working backwards from that, what do you need? Like, you need the high cardinality, you need the high dimensionality and all this stuff. And I feel like that definition really took hold for about three years. In 2019, 2020, maybe 2021, all of the money started rushing into the space, and suddenly, anyone who was doing anything with telemetry was like, cool. We do Observability, too, which, on the one hand, is like, it's a good problem to have. It means that what we were talking about really resonates with people. And at the time, I was naïve enough to think that, oh, well, they're co-opting our marketing language, but surely they're building the same technology under the hood. It's just a matter of time until they release it. I don't believe that anymore.

I think all they did was steal the marketing language, and I don't think they actually have any plans to. I think that, like, Datadog in particular, their business model is centered around having all these different SKUs, right? A different product for metrics, for logs, for tracing, for profiling, for security, and they've got too much money invested in. The problem is that the experience degrades for everyone if nothing connects all these data sources. People are paying to store their data again and again and again and again, but nothing connects it except the engineer who's sitting in the middle just trying to visualize or visually correlate. If that spike is the same as that one, it's fucking broken. My hope is that there will be new startups that are entering the space. So I've kind of given up like, okay, Observability now means, and this makes sense, I'm actually completely on board.

Observability, instead of having a strict technical falsifiable definition, Observability is a property of systems, right? A system can be more less observable if you add some metrics, great, you're more observable. But what we're seeing in the field is that there's a real huge step function difference between, let's call it Observability 1.0, which is about metrics, three pillars, right? And Observability 2.0, which is based on this single source of truth. And it's not just the technology, because o11y 1.0 is very much about MTDR, MTTD reliability, uptime. It's a checkmark before you send your code to production to make sure that it's observable. And Observability 2.0 is about, it's the foundation of the software development lifecycle. It defines your velocity, how fast you can ship, how well you can ship, the quality of what you ship, your ability to iterate quickly, your ability to identify what your customers are actually doing and why, and build on that. It's your ability to see what's happening in the wild and make decisions based on real data and then feed them. Because this is all about feedback loops, right? And it's about learning to be a developer where you're developing with fast feedback loops.

And it's like the difference, o11y 1.0 is about, okay, this is something that you tack onto a product...2.0 is about, this is how you build the product, right? So many teams are stuck in 1.0 land and they're happy with the tools that they have, but the teams that are going to win are the ones that not only adopt 2.0 tooling, but also adopt the 2.0 mindset of this is how we build software. It's like putting your glasses on before you drive down the highway. You can drive a lot faster, you can make better decisions much more quickly. So I feel like right now, the big problem that Honeycomb has from a business perspective is that far too few engineering leaders even understand that 2.0 is possible because you can have a 2.0 mindset. But if you've only ever seen 1.0 tools, it's janky. It's real hard to like...you can only do so much, right? You really need to see 2.0 tooling in order to really...

But it clicks so fast when you do. So that's really our job. For a long time, I was really disappointed that there are still Observability startups starting. They come up, ping, pong, like here and there, everywhere, but they're all 1.0 tools. They're still doing the multiple storage places. My hope is, and I get why, it's because you have to build an entirely different storage layer from the ground up. And very few VCs have the patience for you to do that. They want you to get right to product, market fit and all this stuff. Now that there are more columnar storage engines out there like Snowflake, I don't know...

I'm optimistic, but I'm optimistic over the long run, our model of Observability will win. Even if Honeycomb completely fucks up in the end state is the complexity of our systems is increasingly demanding it. The complexity of people's systems is skyrocketing. You look at the DORA metrics, and I was always kind of like, dude, it's so weird. Like high performing teams, okay, that takes an hour to a day to restore service. But for the bottom like 80% of teams, it takes them a day to a week to restore service from an outage. How? It's because they don't have Observability.

It's because they can't actually see what's going on. They rely on a few people's brains, people who've been there for a long time, who pack a lot of context into their heads, who can try and reason about it using the very limited data sources that they have. That's why it takes so long over and over. Part of the reason we win so many of our POCs is because over and over, our sales engineers, we help you roll it out, and they'll be like, is this an outage over here? We're seeing something wrong. And people will be like, what? Ten minutes later they get paged and they're like, oh, it's just like once you have this feedback loop, you get used to being constant conversation with your code instead of just like shipping and waiting for someone to get paged. At some point in the next hour two year, right. It's all about hooking up this feedback eventually, even if it's ten years from now, the model that we're talking about is the shape that's going to win whether it's us or not because our systems simply demand it. There's no other way to build software at that kind of velocity and scale.

ADRIANA: I completely agree and I think having that conversation where Observability is considered...is baked into like...you're shifting left on Observability basically, right? Were it's like...

CHARITY: Exactly.

ADRIANA: No, it's not the thing that's tacked on at the end per usual. It's the thing that your developers are considering in the beginning that your QAs are using to troubleshoot shit and write trace based tests and that now your SREs are like, "Oh, I've got the information to solve the problem!"

CHARITY: So many of the promises of Agile development and all these SREs and all of these cultural movements, they've never really lived up to their full promise. And I feel like the reason is because it's not just a cultural thing. You have to have the tools that actually make hard problems easy as well. And the feedback loops with metrics and logs are just painful and arduous and relies on so much on manual cross-correlation and heroes jumping into the break. But when you have the right tools, you can just glance at it and see the answer. And it's what unlocks the ability of teams to just be constantly...When I think about modern software development, I think about feature flags which help you separate releases from deploy so you can be deploying small changes constantly.

CHARITY: I think about future flags, I think about Observability, just the ability to see what the fuck is going on at any point. I think about testing in production and I think about, well, canarying. There was one other thing that was on my mind. There's really just a four thing and they all reinforce each other, right? One of them alone is okay, but you get all of them together. And it's a completely different profession than it is in software development, which is kind of still from the shrink wrap era. It's like you're building, if your world while you're building software is your IDE and your tests, that's shrink wrap days. Your world should be production and telemetry. You should spend more time in your production windows than in your IDE windows. That's what modern software development is like I think.

ADRIANA: Yeah, absolutely. And the final point that I wanted to touch upon is you mentioning...having...the data that correlates right? Where you're not just having to figure out how it's stitched together. And tools like open telemetry definitely enable that. But then I guess part of the irony though, is that open telemetry allows you to correlate traces and logs and metrics. But then if your Observability backend doesn't have a way to show that correlation, then you're kind of up a creek too.

CHARITY: So I am so glad that OTel came out when it did so that I think we were able to have a lot of influence on how the data is gathered. You're absolutely right. Part of observability is the presentation of the information. If you don't have the ability to slice and dice, if you don't have the ability to combine, if you don't have that single sort of truth, then you can't really reap the rewards of Observability, even if you captured it. But capturing it the right way is the first step, for sure.

ADRIANA: Yes, absolutely. And so glad that OpenTelemetry has gone officially GA. The specification has gone GA end of 2023. Long time coming. I'm super stoked for that.

CHARITY: It's a big moment in our industry.

ADRIANA: Yeah, and I'm so glad also that so many of the vendors have come together to rally behind it. And it's really not someone trying to flex their muscles over everyone else. It's such a lovely community.

CHARITY: The only lagger is Datadog. People need to keep putting a little bit of shame and pressure on them because they're the only ones who are not playing nice, but everyone else is, which is a tremendous achievement. Huge kudos to Splunk, who's got like 30 engineers working on integrations every day. We would not be where we are without Splunk.

ADRIANA: Yeah, it's so great. It's so great seeing all these innovations, collaborations, and people really genuinely caring for the project.

CHARITY: It's great.

ADRIANA: And on that note, we have come up on time. And thank you so much Charity for coming on geeking out with me today. This was awesome. One item off the podcasting bucket list for me. Always a pleasure to chat with you. And everyone, please don't forget to subscribe, be sure to check out the show notes for additional resources, and connect with us and our guests on social media.

CHARITY: Until next time, peace out and geek out.

ADRIANA: Geeking Out is hosted and produced by me, Adriana Vileela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout. My wonderful editor daughter will edit out any, any stuff. I pay her good money.

CHARITY: How old is your kid?

ADRIANA: She's 15.

CHARITY: Nice.

ADRIANA: That's a good age. Yeah. And she sports right now...she's sporting some really rad pink hair. Last year, she had gone purple, and I just took her to get a cartilage piercing, which I'm like, hey, I have no issue taking you. No issue taking you. I'll look away while it happens. Yeah, it's super fun. Super fun.

CHARITY: I went to college when I was 15, and I felt very adult at the time. And now I look back and I'm like. I was a child. What was I doing?

ADRIANA: You feel so old when you're in high school or like, when you're 15. I remember when I graduated college and I'm like, everyone looks like a baby.

CHARITY: Yeah. Time of rapid change.

ADRIANA: Yeah, for real.