The One Where We Geek Out on Reliability with Ashley Sawatsky of Rootly

Feb 20, 2024

Ask episode

Chapters

Transcript

Episode notes

About our guest:

As a founding member of Shopify's incident response program for nearly 7 years, Ashley Sawatsky led incident communications and processes. Currently, as Senior Incident Response Advocate at Rootly, she consults with tech giants like Canva, Cisco, NVIDIA, and more on incident response strategies.

Find our guest on:

Find us on:

All of our social channels are on bento.me/geekingout
All of Adriana's social channels are on bento.me/adrianamvillela

Show Links:

Additional Links:

Transcript:

ADRIANA: Hey, y'all, welcome to Geeking Out. The podcast about all geeky aspects of software delivery DevOps Observability, reliability and everything in between. I'm your host Adriana Villela, coming to you from Toronto, Canada. And geeking out with me today is Ashley Swatsky of Rootly. Welcome, Ashley.

ASHLEY: Hi. Thank you.

ADRIANA: And where are you calling in from today?

ASHLEY: I am in very snowy Ottawa.

ADRIANA: Yay. I feel your pain. I went to high school in Ottawa, so I remember having to shovel my roof one year. Well, not me. My parents hired someone to shovel the roof. Yeah, there is a lot of snow in Ottawa.

ASHLEY: Yep. After this call, I will be trudging through the snow to pick my six-year-old up from school. And it's a daily battle in the winter.

ADRIANA: Oh, yeah, yeah. Between that and the freezing rain. I remember lots of freezing rain in Ottawa. I'm like, really?

ASHLEY: Lots of that too.

ADRIANA: Yeah. No snow in Toronto. The temperature is just like a little bit above zero. So it's just we get rain, it's like why?

ASHLEY: It's a little sad. I know. Well, I'm excited to come tomorrow and get a little break from the snow, so that'll be nice. I'm only spending. I'm coming. We're going to have dinner and then I'm going to leave in the morning.

ADRIANA: But it'll be a worthwhile trip.

ASHLEY: It will, yeah. Lots of good folks at that dinner, you included. Can't wait. Yeah.

ADRIANA: Excited, excited. All right, well, we are going to start off with, first off, some lightning round questions. Are you ready?

ASHLEY: Okay, I think so.

ADRIANA: Question number one, are you a lefty or a righty?

ASHLEY: I'm mostly a righty, but sort of ambidextrous in some things, like golf. Oh, cool.

ADRIANA: Yeah, I'm a lefty. And so anyone trying to teach me sports that require dominant hand throws people off.

ASHLEY: I think it's because my mom's left handed. So it's like everything my mom taught me how to do, I do left handed.

ADRIANA: That is so cool! I kind of impose my left-handedness at home with the way that I put things on hangers because I do it. Like, people who are right-handed probably don't know this unless they live with the left-handed person, which is like the way in which you orient your jackets when you hang them on a hanger. So yes, I feel you. I am that left-handed person. At least in my house growing up, my mom was left-handed as well. So there were two of us, two against two against the righties. So it was evenly matched.

ASHLEY: I wonder if it's genetic.

ADRIANA: I think it is.

ASHLEY: Interesting. Yeah. Nice.

ADRIANA: Yeah. All right, next question. IPhone or Android?

ASHLEY: iPhone. Die hard iPhone. I can't do the green bubbles. Sorry.

ADRIANA: I know the green bubbles make me a little bit sad. This is why I use Signal or WhatsApp rather than the messages app for non iPhone people.

ASHLEY: We have a joke at Rootly because we love to have a group text going that green bubbles are immediately out. It's totally joking. We absolutely do not screen candidates based on green bubbles. But yeah, it takes some getting used to where I'm a big iPhone user and just Apple in general. As you can tell, I got the AirPods, the whole thing.

ADRIANA: Yeah, I'm a definite ecosystem convert as well. If Apple had a fancy podcasting mic, I would buy it.

ASHLEY: I actually checked if Apple had a mic when I bought the blue Yeti, but I have. It's just the default.

ADRIANA: So sad. Okay, next question. I think I know your answer. Do you prefer Mac, Linux or Windows?

ASHLEY: Yeah, I'm a Mac user. Windows would be a second. Was it KubeCon? One of the conferences we did recently? I'm pretty sure it was our KubeCon merch. I did a Windows 98-inspired sticker sheet that I made with our designer Jerry, and that was our little homage to Windows 98.

ADRIANA: I remember this.

ASHLEY: It was a good one. It was a good one.

ADRIANA: It was like very nostalgic. I saw it and immediately I'm like, yes, it's got the vintage vibes.

ASHLEY: That was like the first operating system where you could customize things a little bit, at least that I knew of. Like you could change the top of your windows to have that little gradient bar. And to me that was just like the most exciting thing ever.

ADRIANA: I feel you. I do enjoy some nice customizations. Okay, next question. Favorite programming language.

ASHLEY: I have a lot of fun with CSS, but I'm going to give a shout out to Ruby. We're a Rails shop at Rootly, and before I worked at Rootley, I worked at Shopify, a massive Ruby monolith. And I just got to know the Ruby community really well. And I think that the community around Ruby is unmatched.

ADRIANA: It is a very vibrant community. Absolutely.

ASHLEY: It's incredible. Yeah.

ADRIANA: I've never known someone who's written code in Ruby to say, "This sucks." Everybody loves, loves, loves Ruby.

ASHLEY: Yeah, it's kind of a love hate. I think some people, when they're new to it, they hate it. But the people who have been programming in Ruby for a long time, if you love Ruby, you will never take a job that's not coding in Ruby.

ADRIANA: Yeah, I have a friend like that actually. We did some Java dev back in the day and now she's like a Ruby Rails developer and she doesn't want anything else. Awesome. I think that's great. It just speaks to the power of the language. Right? Okay, next question. Dev or Ops?

ASHLEY: This feels like a trick question. I'm a technically DevRel, so I feel like I need to say Dev. But I'm going to say Ops. I thrive on the ops side, so I'm going to say Ops.

ADRIANA: All right, next one. Also not a trick question. JSON or YAML?

ASHLEY: Oh God. You know what? We actually had a really crazy incident at Shopify that stemmed from YAML parsing, so I'm going to pick JSON just because I'm still traumatized. I still think about that incident in writing that post-mortem.

ADRIANA: It's the traumas that shape our lives.

ASHLEY: It was harrowing.

ADRIANA: Okay, another question that is slightly...more than slightly controversial. Spaces or tabs?

ASHLEY: Tabs.

ADRIANA: All right, two more questions. Do you prefer to consume content through video or text?

ASHLEY: I'm old school. I like reading text.

ADRIANA: Yeah, same. Yeah, give me a video and I'll probably not read it. I mean, watch it.

ASHLEY: I get distracted. Yeah, I watch it and then I open another tab and then I'm responding to emails and I'm like, wait, what was happening?

ADRIANA: Yeah, exactly.

ASHLEY: But if you're reading, you're reading.

ADRIANA: Yeah, and then if you get distracted, you just scroll back up.

ASHLEY: Yeah, exactly. I would even go a step further and say, ideally, print. I actually just ordered a print copy of the Site Reliability book, like the one, the Google one that Jenn and Niall did and all the people who contributed. And I've been working my way through the actual print copy of it. It's nice. I can highlight it with a real highlighter. It's so good.

ADRIANA: Yeah, there's something very nice about that. Sort of very interactive, tactile aspect of having a print book.

ASHLEY: Yeah.

ADRIANA: My only thing with print books, I like them, but most of my books now are ebooks because I simply do not have the room in my house.

ASHLEY: Yeah, that is becoming a problem for us. Our bookshelf is overflowing. We probably need another one. But I also like it because your computer, your phone, everything, you have so many apps, so it's very easy to get a notification. And then I'm distracted and I'm responding to emails and I'm like, wait, I was trying to read something like, what happened to my attention span?

ADRIANA: Yeah, totally.

ASHLEY: So I try to, if I'm going to read, set a 25 minutes timer or whatever and just actually pick up a book because it's the best way for me to not get distracted because I have a short attention span.

ADRIANA: Yeah, I feel you. I suffer also from a short attention span. For me, what has worked...not as nice as the tactile feeling of a book...but having a Kindle where that is all it does. So I have that with my breakfast. I'll have my Kindle book on me and just chill for 20 minutes before the day starts. And it's awesome.

ASHLEY: Yeah, I should get a Kindle. That's honestly a great idea. I felt like I didn't need one because I have an iPad. But then you have the same problem. It's just like a big iPhone.

ADRIANA: Exactly, yeah, yeah, I know. I remember when I was actually looking for a Kindle initially, and someone's like, just get an iPad. I'm like, no, I cannot have the distractions. I just want the one thing.

ASHLEY: See, this is the hold that Apple has on me. It's a problem.

ADRIANA: Feel you. I feel you. Okay, final question. What is your superpower?

ASHLEY: My superpower? Oh, I like that question. I think my superpower is that I'm very scrappy. Everything I know and everything that I've done in my career has been pretty unconventional. I do not have a traditional computer science degree. Everything I've learned, I've just kind of learned through watching and doing and figuring it out and asking questions. So, yeah, I think that has been helpful for me in my life, and that's something I will continue doing.

ADRIANA: I love that. I think scrappiness is very important, especially in our industry.

ASHLEY: Yeah, I think it also just builds a lot of confidence once you realize you can actually just figure this stuff out. I think for a long time, it felt like there was some secret trove of information that people had that I didn't or these invisible barriers that existed. And then at some point, you realize you can figure it out. There's no secret. Everyone's literally just figuring it out also.

ADRIANA: Yeah, exactly. I think that's the most comforting thing, is realizing that you're not in this alone. Like, chances are other people are just wading their way through the plethora of information and trying to sort this out. And then if we let each other know that we're all kind of trying to figure this stuff out together, then we can provide each other support.

ASHLEY: Totally. Yeah. Once I got lucky to work at some very well known companies, I worked at Disney and Shopify, and I got to work closely with people I really admired in those jobs and our execs and leaders. And once you realize, of course, they've got so much experience behind them, and that's what gives them the ability to figure out the next really difficult challenge. But they're still figuring it out. They are sometimes unsure of what to do. And once I saw that up close, I was like, oh, my God, I never would have thought that these people are also just kind of making the best decision that they can and hoping it works and adjusting as they go.

ADRIANA: That's true. Yeah. It's like the, we're all human at the end of the day.

ASHLEY: Yeah. And especially tech. It changes so much. No one knows exactly what's coming next and what's going to work. You just have to be willing to try things.

ADRIANA: I guess that's true. Yeah. Expert one day and newbie the next, right? Pretty much. So I think this is a good springboard into our main discussion. So you mentioned that your superpower is being scrappy and that you don't come from a traditional comp sci background. What is your background?

ASHLEY: I'm like, it depends how far back you want to start. I think I'll start at Disney because it was my first tech job. So I worked in a part of Disney that tragically is now defunct, called Disney Interactive. And it was like the tech products division of Disney Consumer products. So it was like apps, online gaming, websites, and digital products. And I started out on the tech support side and eventually kind of like, fell into this weird role that was communications focused, that was handling special cases, basically things that had happened where somebody's experience with Disney was not optimal. They've gone through the support team, they have not gotten the resolution they need, and they've filed a case. And I would deal with those cases and talk to people and try to just make it better, make it magical.

And I had a lot of fun doing that. And that was like one of the first times, I think, in my professional life that I had discovered something I felt like I just kind of had a knack for. So I was spending a lot of my time fixing things that were broken, but they were more experiences, they're more technical things, too. They bought a game and it's not working, and we've got to file a bug report, but it spanned a lot of different things. And so that was when I was, ah, I kind of like having hard conversations and solving problems, and it was probably one of the first times I had really felt challenged in a role, too. And I did that for some time. And then a friend of mine - shoutout to Colin, who I worked with at Disney - he had recently left and he had joined this company called Shopify. And he was like, this is so cool. You need to join Shopify. It's so much fun. I was like, it sounds like a weird multilevel marketing scheme. Like, what are you talking about? This was in 2015. He was like, you get to work from home. They send you a laptop. And I was like, that's a scam. Turns out now Shopify is very well known and is, in fact, not a scam. It is a very real global software company. So all that to say, I joined Shopify and I was a founding member of this incident response team that they were building out. And this was very early on in incident response at Shopify. So it didn't actually have much of a technical focus. It was a bit more focused on the customer support side of things, where, again, something had gone very wrong and we were reacting to it, trying to make it right. If something had gotten to the three of us that existed in incident response, it was because they had gone to an executive or had gone public with some issue, or we made a big mistake and it needed fixing. And so as we were building that out, we started realizing that a lot of the solutions for this problem should be solved at the support level, because you can't have three people responsible for everything that goes wrong.

It didn't make sense for this to be an escalation to a separate team. We thought anybody should be able to handle something like this, a customer issue, and fix it. So we focused a lot on that enablement side, like how to have hard conversations. What are the rules? When do we give a refund? None of that was defined. And then, as we had sort of to say, worked ourselves out of that job of being the escalated customer support, we ended up getting a little closer to the resiliency engineering side. And this is where my real incident response sort of career was born. There was no connection between what was going on with the platform, whether it be like a technical issue or an outage, and how we were communicating externally. So that was like the first task that I had was to build sort of a bridge between engineering and communications and customer support and social media and all of that to say, like, when a super technical incident is going on in incident room and everyone's looking and nobody knows what it means, here's how we communicate about that externally and internally.

Here's how we tell support what's going on. So it was very focused on building processes, building communications that took that technical stuff and made it make sense outside of it, because Shopify is a very technical product with mostly a non-technical audience. And that is when I got really interested in the technical side and reliability and what was actually happening with a platform and how our infrastructure worked and what it meant to be in the cloud. And it was my first introduction to, oh, there is physical data centers and how does the Internet work? And from there, I just dove deeper and deeper and deeper into that and learned a lot about how giant, complex system works. And I'm by no means an expert in infrastructure, but I've gotten to learn a lot about it, and that's what sparked that interest. And I've just continued learning from there, I guess. Oh, and then I landed at Rootly, so that's where I am now. I won't skip that part.

I'm now a Reliability Advocate at rootly, so I get to help other companies solve similar problems that I've solved in my career and talk about reliability and incident response and things that I've learned and meet other people who are interested in it and just kind of build a bit of a community around that space. That's so cool.

ADRIANA: And it's interesting, too, because all of the previous experience that you'd had had brought you to where you are now.

ASHLEY: Right.

ADRIANA: I think even the stuff that you were doing at Disney kind of gave you that empathy for the customer that is so important when it comes to reliability that we don't talk about enough.

ASHLEY: Yeah, I think even if I go before that, it's one of those things where, in hindsight, all of these things that I've done started to make sense. Like when I was a server at a restaurant, I was always the girl who would go to the angry tables and help move things over and comp the meal or talk to them, talk them out of leaving us a bad Yelp review. And I sort of found at some point that combined with software, and it was like this whole new world opened up where that was still a thing. But there was also a lot of new stuff for me to learn in terms of how complex systems work and the infrastructure that powers the Internet and apps. And I just found that so interesting that I couldn't stop learning about it.

ADRIANA: Yeah, it's ridiculously complex, but it's also incredible to realize that even the job that you mentioned as being a server and having to deal with angry customers, I mean, there is no better test bed for being in reliability than to deal with angry restaurant customers because that can be really scary.

ASHLEY: Yeah. And I realized that that had value that I didn't always see at first. I think I saw that as sort of a, like, that's my former life from before I had a real job in tech and everything. And then at some point I realized not everyone has that experience. Not everybody's had to do that all night and talk to people face to face who are mad at you. And that builds a lot of communication skills that not everybody has. So I found that that was something that I could bring into a world that I was very new in, like tech, and still have something unique that I was bringing to the table. And then eventually I kind of transitioned out of that more customer focused side to operational and more technical as I went along. But that continues to be very useful. I don't think there's a situation in life where the ability to talk to people stops being helpful.

ADRIANA: Yeah, that's so true. Yeah. And I think also there's this misconception that in tech you don't need to have those soft skills, especially if you're like a software engineer, for example, that as long as you can code, then that's all that matters, or whatever it can be. Even for an ops person, as long as you make sure that the systems are up and running, it's all good. But it's not. You have to be able to communicate as a software engineer. You have to be able to communicate your ideas beyond just the code. As an operator, you have to be able to also communicate your ideas beyond just operating those environments.

And I wish there was a little bit more emphasis put in communication in our education system because we all like, you know, I, when I went to school, we had to take a technical writing course and everybody just freaking groaned at having to take this technical writing course. And I hated it too. It was so dry. But it was useful, right, because you need an effect... Technical writing teaches you to communicate in a very effective and efficient manner, which is important in our industry.

ASHLEY: Yeah. And then if you look at reliability and incident response specifically, it becomes even more important against the backdrop of those situations and the pressure that people are under. And the technical skills are also important because those are some of the most technically complex situations you run up against. That in combination with somebody might feel some type of way about what's broken. And it could be your customers, but it can also be your support team and your marketing team that had a big launch that day and the exec team that doesn't understand infrastructure, but wants to know in vivid detail exactly what's happening, but in like ten words or less. And you're like, "AH!"

ADRIANA: And that is a skill. That is a skill. Speaking executive versus speaking to your peers or to your manager. Like, it's different language altogether.

ASHLEY: Oh, yeah, I wrote a blog post about that recently. That's something I learned a lot about in my time, especially at Shopify, working with execs and managing incidents and realizing like, oh, yeah, you need to be very intentional. And it's not because execs are mean and scary. It's because they're looking at things from a totally different vantage point. They understand the business differently. They have so much context that you don't, and they have very little time. Their time is very accounted for, very expensive. You need to learn how to be effective.

ADRIANA: Yeah, and the other thing that I learned was being able to speak in dollars and cents also goes a very long way with execs because they want to know, like, yeah, this is great, but what do I get out of this thing? So you want to buy this new whatever. So what?

ASHLEY: What do you need from me? How much is it going to cost?

ADRIANA: How is it going to help in dollars and cents?

ASHLEY: And, yeah, exactly.

ADRIANA: I feel you. Another topic I wanted to touch upon because you mentioned that you worked in the reliability space at Shopify. When you and I chatted earlier, you'd mentioned that you'd been on-call. Can you share some of your on-call experiences with folks?

ASHLEY: Yeah, I have been very on-call for a long time. Like I said, it was just the three of us in kind of the earliest days of instant response, and we just threw ourselves on a pager thinking this will be like the easiest way for anybody to get a hold of us. And there were a lot of late night wake ups, and we had very little process built around what to do when the pager goes off. It was just kind of, if you get paged, you figure it out. And eventually that scaled. And we learned to manage expectations, manage what qualifies as a page versus like, send me a Slack ping and I'll deal with it in the morning. Maybe something that really comes to mind as a very intense version of the on-call experience was the Black Friday Cyber Monday preparation cycle and weekend at Shopify. Just because ecommerce is the highest pressure weekend of the year, and I think every Black Friday from 2016 or 17 to 2022, I spent 96 consecutive hours on that Black Friday pager ready at any moment.

And I learned a lot, honestly, I think it's necessary to put yourself on a pager even if you're on operations at some point, because when you're building all of those processes and plans of what's going to happen, when you know that it's you that's going to get paged, you care so much more and you're a lot more thoughtful and you have the experience and context to say, like, yeah, this thing looks good on paper, but that is not going to work. I think that's a common mistake that I see in incident management and process, where people want to prepare for a specific scenario with a specific playbook that they will then execute. And sadly, it just almost never happens that way. So people overrotate to process a little bit and think, like, we want to get to a point where you can blindly follow this process when something happens, but that's just not the reality. So I think the most important thing, or maybe I don't know if it's the most important thing, but I think any company that has a pager should invest more time into talking about the on-call culture at their company. What it means to be on-call, what's expected of you and what's not expected of you. Should you be glued to your laptop for that entire time? Can you go walk your dog? In my opinion, you should be able to do that. You should have reasonable expectations.

Like, get to your laptop within 15 minutes, respond to the page within five is a good benchmark example. But, yeah, I could talk about on-call for a long time because I've done so much of it. And at Rootley, that's like one of the biggest things that we're on a mission to do is just make that experience better for people.

ADRIANA: Yeah, because, I mean, it can cause some serious PTSD.

ASHLEY: It can. It can cause very real stress and in some cases, probably even actual trauma. I had a great experience. Shopify is an amazing place to work. It's a great culture. So I won't say that it was traumatizing, but it was stressful, for sure. And those of us who went through those sort of early days of it, once we had built up that empathy, we put a lot of effort into onboarding to prepare people for what it's like. But you also kind of want to balance that with making it better.

So we would talk a lot about on-call health, and especially in the lead up to major events like Black Friday, we would have a lot of messaging around on-call and wellness. Like, are you checking in with yourself every hour to make sure you have eaten and you have taken a screen break and you've gone on a walk, or do you have a bottle of water around? Just those little things that can remind people. And that's something that I've actually encouraged some customers that use Rootly to build into some of their automation for incident commanders because we do have a little prompt that can pop up when you're assigned a role in an incident. So say you're assigned incident commander. It might say, here's your responsibility. But I usually encourage people also say, put something encouraging in there to say, take a screen break. If you have to lean on your secondary on-call, you're not in it alone. And that's another thing I feel strongly about, is it should be illegal to have an on-call rotation with no secondary.

ADRIANA: Yeah, absolutely. Because that way you feel like you're supported. Like if shit hits the fan, you know that you can lean on someone else.

ASHLEY: Yeah. And life happens. Ideally, for an on-call shift, you're available, you're on-call, but what happens if your dog breaks its leg and you got to go to the vet? Things happen and you need to have some sort of backup plan. Single points of failure are bad in software, and they're bad in people systems, too.

ADRIANA: Absolutely. And you touched on something that I thought was so important, which is really building up that culture around incident management, because, as you pointed out, you can try as hard as you can to account for every single little thing that will happen with your system, but reality strikes, and you're mostly dealing with those unknown unknowns rather than the known unknowns. And so I think being able to mentally prepare for it, and I guess being also in a psychologically safe place where you can actually troubleshoot in peace is super.

ASHLEY: Yes, totally. Yeah. I think there's, like, an element of protection that SREs need, and I feel pretty strongly about having not just SRE commanding the incident and also trying to fix it, but having an incident commander whose job it's not to fix what's broken, it's to protect the responders and keep things on track and keep those distractions, like that exec that storms into the channel, that's like, the incident commander should be like, whoa, let me stop you there. I'm going to field this. You're not getting anywhere near our on-call engineers. They're working on the problem. It's not their job to explain to you why this happened when we haven't even mitigated the problem yet, let alone found a root cause, which is like a whole other thing but people who don't understand the technical aspects, which is fine, you don't have to understand it, but there's kind of that healthy boundary and respect to say we're not there yet. We're currently investigating the problem, and here's where you can get an update every 15 minutes, and it's not in this SRE's DM, so back it up a little.

ADRIANA: Yeah, as you said that I was getting flashbacks to earlier in my career of being on a call during a major issue where there's some friggin exec who's, like, poking their nose into your business and, oh, well, I used to code 20 years ago, why don't you restart the database? And it's like, buddy, back off.

ASHLEY: Yeah, you really get it from all sides. And incident response, too. I mean, I've seen outages where you get the people coming out of the weeds on Twitter...X, or whatever. I was in it for eight years, and I could have resolved this in five minutes, and you're like, please.

ADRIANA: Get off my back, buddy.

ASHLEY: Yeah, totally. It's wild. It's a lot of pressure. So I think companies owe something to the people who are there solving some of the worst, most urgent problems to make sure that there's a culture and process and tooling in place, that it's not harder than it needs to be because it's already pretty hard, even with all that stuff.

ADRIANA: Yeah, exactly. Like, you're stressed, you're in the middle of an incident, there be problems.

ASHLEY: If you don't want to burn through your SREs, then you got to make the experience bearable.

ADRIANA: Yeah, absolutely.

ASHLEY: Sorry. No, you're good.

ADRIANA: Yeah, I was going to say, I remember my previous job. I was managing a couple of teams, and one of the teams I managed was a platform team, and we had an on-call rotation because we're managing the Hashicorp infrastructure. And part of it included, like, I had a sub team of SREs and one of the guys on the team, he had been in operations for most of his career and had had some very traumatic on-call experiences. So even the thought of him being on-call, he was very much not down for it. And I couldn't blame him for it either, because that is some deep-seeded trauma that can be very hard to get rid of, to overcome. And I don't know, maybe being in a more welcoming environment can help them heal, but maybe there are some wounds that are just too deep where you just kind of have to avoid those types of roles, if you've been in such a traumatic spot.

ASHLEY: Yeah, I'm sure that does happen. I will put a positive spin on it in that I have kind of a nice story, I just remembered, about an example of that culture really existing. And for a time when I was at Shopify, our CTO was Alan Leinwand. He's now the CTO at Webflow, and they just so happen to be a customer of ours. But that's not why I'm shouting him out. I am shouting him out because I vividly remember an incident at Shopify where the engineer who caused the incident, it was just...you know...he had shipped a PR that broke something. It happens. And he felt so bad and he was in the incident room channel, which...Shopify is a big company.

There's thousands of people in this channel watching this. And you can tell he's flustered and he's embarrassed and he's saying, like, I'm so sorry. I should have tested against this condition and I didn't, and I'm so sorry. Know, you guys are all having to deal with this, and I'll stay late and blah, blah, blah. And Alan, who I know was always paying attention to what's going on in incident room and had a large amount of trust for the engineering team to handle things. He wouldn't jump in and start bombarding everybody, but he just dropped a message, a very discreet threaded response on that engineer's message that just said, like, "Hey, it's okay. You did the right thing. You noticed something was broken.

You paged the on-call team and every great engineer has broken things. It's not what you break, it's how you fix it." And just gave him a really nice reassuring...and didn't make a whole "@here I'm the CTO. Look at me. Praising." It was just tucked away in a thread, just like some words of encouragement and reassuring him, this is totally normal and it happens and you're fine. And I think that was just like such a nice example of how the culture can be if you actually have people who understand instant response and just have empathy for people. Because engineering is really freaking hard. That was nice.

ADRIANA: It's such a nice story. I love it so much.

ASHLEY: So shout out Alan. Great CTO, in my experience.

ADRIANA: We need more folks like him. Absolutely.

ASHLEY: Exactly.

ADRIANA: One other thing that I wanted to ask, because now you've gone from Shopify, big huge company, to Rootly, startup, how has it been? What do you notice in terms of going from a really large company to a really small company?

ASHLEY: Yeah. Oh, my gosh. It's been crazy. It's a huge adjustment. When I had left Shopify, I didn't really know what I was going to do. I didn't have much of a plan. And JJ, our CEO and co-founder at Rootley, approached me and was telling me about the company and know role he wanted to build. That was kind of a developer relations-style role, but really focused on reliability and incident response. And I'd never been a DevRel. I didn't even frankly fully know what that was. But he wanted somebody who had been on-call and who had done the work. And a just hearing his story of how the product came to be, that he was solving similar problems at Instacart with our co-founder Quentin, who was their first SRE, I just felt like he really understood the reality of what it was like to work in incidents. And when I looked at the product, I really just loved the product. I, in my previous role at Shopify, had looked at incident management tooling. This was a while before Rootly had come on the market. And spoiler alert, we built our own because we just couldn't quite find anything that fit what we had wanted at the time.

And when I saw Rootly, I was like, finally someone gets like, this is what I was looking for, not some big, clunky, over-engineered standalone platform that I'm never going to be able to get anybody to adopt. That's going to take months of development work to just even get up and running. This is so simple. It just plugs in. We can use it in Slack. So that was the first thing for me, was just really liking the product in terms of transitioning to a startup. Those first three or four weeks, I was like, oh my God, I don't know if I'm going to make it. The adjustment of the pace was crazy.

I just couldn't believe how fast they were shipping. I thought I worked fast in my previous jobs. I thought we had a fast pace. That is nothing compared to a true Series A startup grind. But once I realized that it wasn't impossible, I just had to shake off some of the big corporate rust and stuff that I had in my system. I'm like, well, what do you mean? We're not running this through five different teams for approval. You trust me to just do it and ship it? And it was like, yeah, if you think it's going to be cool, just ship it. So it was just a massive increase in the amount of autonomy and the pace I was working at and also the amount of creativity.

I think that's like one of my favorite things is we don't have the brand guidelines that you have to adhere to. And this is what we say and what we don't say, and this is how much we spend on this. When you're at a really large scale, there's a lot of process and rules, and they're established for good reason, because when you have 15,000 people, you got to have them marching in the same direction or else it's going to be a disaster. When you have 25 people, you can be really tightly aligned without all of that friction. So that was just like a breath of fresh air. And now I feel like I've really hit my stride with it and I'm less scared to move as fast as we move. It was a little scary at first. It was like I was standing on like a freeway and cars are just like whizzing past me and I was like...AHHHH!

ADRIANA: And now you found your groove?

ASHLEY: Yeah, now I'm good. I'm having a lot of fun. It's a great team. Everybody cares so much and is so fun and passionate about what we're doing. And a lot of people have experience doing this, too, like Ryan, who is on our post-sales team. He has done similar jobs to what I did at Shopify, but he was at Twilio and these other companies. So having people you can really geek out about, what was it like building incident response at a hyper growth company? And you're like, oh, my God, this was so hard. And this was so fun and I learned this. It's just a really energizing group to be a part of. So it's super fun.

ADRIANA: That's awesome. Well, thanks for sharing. Well, we are coming up time. And before we go, do you have any parting words of wisdom on incident response or generally anything tech related that you would like to share with our audience?

ASHLEY: My parting words are, if you are curious about incident response, becoming an incident commander but you are scared to get started, I really recommend you reach out to somebody who's doing it well at your company and ask if you can just shadow them and watch what they're doing and learn from them because it's not as scary as it looks, and you can totally do it if it's something you're interested in. And you don't even have to be all that technical because I sure wasn't. You can just learn. And of course, I have to give my little Rootly plug. If you want to learn how to make life better for your on-call responders and just streamline your entire incident response process, check out Rootly. We automate incident management in Slack across tons of different integrations. Whatever it is you use, we integrate with it and it just makes managing incidents so much easier. So check us out.

We're at rootly.com. We'll give you a free demo. It's free to try. You could even set up a trial, no credit card. Start playing around with it. And we do tons of events. So if you're heading to KubeCon, maybe in March in Paris, I'll see you there at the Rootly booth. I don't know. I'll be there.

ADRIANA: Well, thank you so much Ashley, for geeking out with me today. Y'all don't forget, subscribe and be sure to check out the show notes for additional resources and to connect with us and our guests on social media. Until next time...

ASHLEY: Peace out and geek out.

ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. It be sure to follow us on all the socials by going to bento.me/geekingout