The Ultimate Way to Handle Deploys on a Medium-Sized Team

Joël discovered Bardcore. Stephanie planned and executed an IRL meetup for folks in the WNB.rb virtual community group in Chicago and had a consulting win.

Together, they discuss what deployment processes look like for clients in their current workloads.

---

This episode is brought to you by Airbrake. Visit Frictionless error monitoring and performance insight for your app stack.

---

Hildegard von Blingen YouTube Channel
Hildegard von Bingen - Historical Character
WNB.rb
git flow

Transcript:

STEPHANIE: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Stephanie Minn.

JOËL: And I'm Joël Quenneville. And together, we're here to share a bit of what we've learned along the way.

STEPHANIE: So, Joël, what's new in your world?

JOËL: I've been getting into something that's kind of fun and quirky. It's a new musical genre called Bardcore.

STEPHANIE: Bardcore.

JOËL: Yes, it's basically re-mixing pop songs to make them vaguely more medieval, oftentimes using acoustic instruments, something that sounds a little bit more like maybe a lute in some flutes, oftentimes also kind of changing the lyrics a little bit to use more old-timey language. When the lyrics use words from modern life, sometimes changing them to something that would fit more in a more medieval setting. It's a lot of fun.

STEPHANIE: That sounds so fun. When are you normally in the mood to listen to Bardcore?

JOËL: It can be fun while coding because it's fairly chill as a genre. Honestly, I feel like it can also be good when I'm just sort of feeling a little bit nostalgic or daydreamy. I think it's good for that mood as well.

STEPHANIE: I love that. I can't wait to go and listen to some Bardcore after this.

JOËL: Let me recommend the YouTube channel Hildegard von Blingin' as a great entry point into the genre.

STEPHANIE: Incredible. I can't wait. In fact, I'm going to end up sharing it with all of my D&D friends too.

[laughter]

JOËL: The channel is a play on words of an actual historical character, Hildegard von Bingen, who is, I want to say, a 12th-century nun but also a polymath. So she wrote on all sorts of topics, from biology and the natural world to theology. She was a musician, just one of these like really talented people that made a mark on the medieval world. So it's kind of fun that they used her name as the inspiration for the channel.

STEPHANIE: Yeah, that sounds right up your alley. I knew that you were going to come with a historical tidbit about Bardcore. [laughs]

JOËL: Another really cool thing that I appreciate about the channel is it's not just audio. It's also beautifully illustrated. So the creator has created visuals inspired by medieval manuscripts illustrating the contents of the song. So it's kind of funny to see something that...modern pop songs aren't always the most deep lyrics, and to see them given this medieval manuscript treatment is amusing, for me at least.

STEPHANIE: That sounds really funny and also kind of calming. I was just thinking about what you said earlier about how they sometimes rewrite the lyrics to be more about medieval life. And I love the idea of taking the things that pop songs are about these days and applying them to historic life back in the day.

JOËL: A lot of pop songs also are about love, and romance, and breakups. I think that kind of fits with some of that 12th-century troubadour-style romantic songs because that was definitely the kind of thing that they were singing about back then as well.

STEPHANIE: Absolutely.

JOËL: So I've been jamming out to Bardcore this week. What is new in your world, Stephanie?

STEPHANIE: I'm really excited to share that I had an awesome weekend. One of the things that I had been doing the past couple of weeks was planning an IRL meetup for some folks in Chicago, people in the WNB.rb virtual community group. I've mentioned it on the podcast before when I was a guest. But WNB is a Ruby community group for women and non-binary folks. And we just started creating regional Slack channels.

And so I started a little Chicago channel and planned a brunch. So on Sunday, a few of us, I think it was six, some old friends and some new met up for brunch in Logan Square in Chicago. And it was really awesome to do a local meetup. I haven't done something like that since pre-COVID times, and so it felt really special.

JOËL: That's exciting. Were you big into the meetup scene pre-COVID?

STEPHANIE: So I was working remotely for a previous company when I moved back to Chicago, and so was still trying to meet people here, find a community, find some friends. And I did go to a few community groups, but that was not too soon before COVID started, and so I didn't get to really invest in them the way that I had hoped. So it's really exciting to me to potentially be able to start doing that again.

JOËL: This new meetup that you were at, was it focused more on the social aspect of things, or was it a more technical meetup?

STEPHANIE: It was definitely more of a social aspect. I would be really curious to know if that group would want to focus on some more technical things. But we had a nice diversity and experience levels and the types of work we were doing. So there were a few of us who were consultants, a few of us at product companies. And I think we shared a lot about our different experiences. We talked a bit about the pros and cons of product versus consulting.

And so it was really nice to just learn more about what other people are up to, what tech and framework people are using, and chat casually in that sense. But I also definitely see some more opportunity to focus on technical stuff if that moves us.

JOËL: I think that was probably my favorite part of Ruby meetups back when I was attending those a lot here in Boston, where I'm based, getting to chat with other developers in the city, hearing about their experiences on different topics. And oftentimes, it will sort of revolve around tech to a certain extent, but it's not always like a formal have a presentation. Sometimes just socializing is almost more fun or brings more value to me.

STEPHANIE: Yeah, I totally agree. I also wanted to share another thing that happened to me this week. It was a bit of a consulting win. So on my client project, we have been having retros every two weeks at the end of the sprint. But I was noticing with a fellow thoughtboter that we weren't really getting a lot of engagement in retros. It was kind of the same folks speaking and bringing up issues because we were doing it in a style that was like a retro board, and then folks could write in cards or raise their hand in the meeting to add something to one of the columns.

And so, we ended up proposing to do a round-robin style format for retro. And we just had our first one yesterday using that new format, and it was received really well. Everyone went around and shared things that went well. And then, we went around again and shared things to improve or risks or concerns that we had about the sprint. And it was really nice to have everyone participate, to hear folks piggybacking off of what other people said. And I think we were able to get a better sense of what the group was feeling.

And yeah, there was a new hire who was just observing our retro, and she is going to be facilitating these kinds of meetings for other teams. And she seemed really into it and wanted to bring it over to other teams as well and try it there. And so that felt really good to know that we were able to make a change that was an improvement for our team but might even have an impact on other teams at the company as well.

JOËL: I love that. I think a lot of what we often bring to the table, because we've seen things at a lot of different companies, is not just code improvements but also process improvements. Every company is different, so you can't always just copy-paste things from one place to another. But being willing to try new things, experiment, and then follow this iterative, continuous improvement approach, not just with the code but with the process as well, I think, is something that is really valuable in the work that we do for our clients.

STEPHANIE: Yeah, absolutely.

JOËL: And it sounds like here you iterated on their retro process. And everybody seems to really like this new iteration, so that sounds like a big win. Congratulations.

STEPHANIE: Thanks. I really appreciated that they were open to trying. That made me feel really good and makes me feel empowered as a consultant to be able to like you're saying, leverage that experience and suggest things that can just improve the quality of life for our clients.

JOËL: Another area that I think we've seen a lot of different ways of doing things, and we've actually been able to iterate a lot as far as process goes, is deployments. How do we get our code from, let's say, passes code review, and then, at some point, we want it to go live in production? So what does deployment look like on your current client, Stephanie?

STEPHANIE: I'm glad you asked because I'm experiencing a deployment process on this client that's actually a bit different than what I have seen before. So this client is not a super big team, but maybe, I don't know, between 30 or 50 engineers would be my guess. I am working on a smaller team with just four developers. And so I'm seeing a lot of code get merged into our big Rails app pretty frequently by other teams. And we are also merging to the same app.

So my client has release managers who rotate each day and go through all of the different teams' pull requests that are ready to be merged. They will merge those pull requests on the developers' behalf. And then once everything is merged into an integration branch, they will then merge all of that stuff into their production branch and kick off a deploy.

JOËL: Wow. So does that mean that developers on your team don't merge their code? You just when you get an approval, you ping the release manager and ask them to merge it for you?

STEPHANIE: Yeah, so developers don't merge their own code. We might move the card into ready for deployment, and that's how release managers know that that PR is ready to be merged.

JOËL: And are you then following something that's roughly like Git flow where you've got this sort of development branch, and then at some point, commits get maybe cherry-picked over to the main branch, which then gets released? Or maybe it's even a special dedicated release branch. What does that look like in terms of the Git workflow?

STEPHANIE: Yeah, we have that release branch that you mentioned that eventually gets merged, either through the GitHub GUI or a CLI by the release manager, into the main branch, essentially. And that's what then gets deployed.

JOËL: How do you handle situations where a feature goes out to production, and then you realize that there's a bug or there's something that you don't like about it, and you would like to revert that feature?

STEPHANIE: Yeah, that's a great question. This has happened to me once now, where I merged some code that ended up introducing a regression. And unfortunately, I wasn't tagged or pinged, so I didn't really know about this until the next business day and caught up with Slack and saw that someone else had to resolve my issue, which was kind of a bummer, I think, because with this process, once that code is, quote, unquote, "done," since I'm not the one merging it, and I'm not the one deploying it, I don't get a chance to follow up on the changes in production and then check to see if they look good.

When things go wrong, it seems like it kind of takes a bit of time to figure out how to get it resolved like; who would have the context? And then, if they're not available, someone else might have to jump in and fix it. So it's been interesting because, on one hand, I totally understand that they want to be releasing just once a day. Like, it's nice to have a dedicated person do all of this stuff that is work and would take away time from normal development.

But I do sometimes feel like I don't have as much ownership over my feature with this process because, like I said, it just kind of is out of my hands. And oftentimes, I might be done with my work, but that doesn't get deployed for a few days depending on other things going on with the team.

JOËL: That's interesting that you mentioned that it might not be deployed for a few days even though it's done and maybe merged. I think, generally, we assume that merging a commit into the main branch and deploying it are going to be more or less the same thing.

But oftentimes, you might end up in a situation where there's a feature that's done in development, but we don't want it to actually go live for our customers for a while yet. And that might be for technical reasons because we're waiting for other pieces to be in place, or it might be for business reasons because we did the work, but this feature has to come out on a particular date, and so that's when it's going to go live.

So then you end up in that awkward situation, maybe where you want to deploy something else. But you've got a commit already on the master branch that can't go out with the others. And you've got to do an awkward cherry-pick. Have you ever been in that situation?

STEPHANIE: I have. I remember being on a project where we had features in our main branch, but that hadn't been deployed to users yet. We actually didn't want that to be live yet but then had an issue with an existing feature that was already live that we had to make a hotfix for. And that was definitely one of those cherry-picking situations that did become a bit hairy and wasn't too fun. It sounds like you have had experience with that type of deployment process as well.

JOËL: Yes, I think of a project where that was a very common problem because there were a lot of features on that project that were gated to a particular time. So a lot of the features going live for customers were decoupled to the actual development lifecycle. And on that particular project, we used a lot of feature flags on the commits. So we'd control whether or not a feature was live for the customer. It wasn't, is this commit in the main branch, but it was, is this feature flag on or off?

STEPHANIE: Yeah, we're using feature flags on this client project as well. And so, in some ways, I think that if we did have a more continuous deployment process, it would be okay because this big feature that I'm working on on my team we're not trying to go live until a month from now, but we have been slowly, incrementally pushing features underneath the flag. But even then, we do still have a bit of an async process because of this daily release flow.

MID-ROLL AD:

Debugging errors can be a developer’s worst nightmare...but it doesn’t have to be. Airbrake is an award-winning error monitoring, performance, and deployment tracking tool created by developers for developers that can actually help cut your debugging time in half.

So why do developers love Airbrake? It has all of the information that web developers need to monitor their application - including error management, performance insights, and deploy tracking!

Airbrake’s debugging tool catches all of your project errors, intelligently groups them, and points you to the issue in the code so you can quickly fix the bug before customers are impacted.

In addition to stellar error monitoring, Airbrake’s lightweight APM helps developers to track the performance and availability of their application through metrics like HTTP requests, response times, error occurrences, and user satisfaction.

Finally, Airbrake Deploy Tracking helps developers track trends, fix bad deploys, and improve code quality.

Since 2008, Airbrake has been a staple in the Ruby community and has grown to cover all major programming languages. Airbrake seamlessly integrates with your favorite apps to include modern features like single sign-on and SDK-based installation. From testing to production, Airbrake notifiers have your back.

Your time is valuable, so why waste it combing through logs, waiting for user reports, or retrofitting other tools to monitor your application? You literally have nothing to lose. Head on over to airbrake.io/try/bikeshed to create your FREE developer account today!

JOËL: How do you feel about continuous deployment in general?

STEPHANIE: So the reason why I found the way I've been describing so surprising is because I am more used to a continuous deployment flow. When I used to work for a product company, and we were a team of, I don't know, like 30 engineers, we'd merge our own work. And then our main branch would automatically be deployed. And so we could make sure our changes looked good in production and then feel a sense of we finished this feature. But we did also run into problems there because our CI build time which had to run for every single time that code was merged into main, took maybe 20-25 minutes.

So whenever we merged to main, we would have to wait for CI to build, wait for the deploy to go through, and that might be a few extra minutes depending, and then confirm our changes. And that became more of an issue when there was a backup in the queue of a lot of people trying to merge code. And it's funny because we kind of want to be constantly merging. That's kind of a sign that things are moving along.

And it ended up being that deployment was the bottleneck in some instances, especially if there was a CI build that broke, and then it was kind of like a car crash a little bit where there was this huge backup. That wasn't great either because when you have to babysit your deploy like that, I didn't find that I had a ton of focus time to go and pivot to something else. I was just keeping an eye on things the whole time.

JOËL: Would you have preferred a workflow that maybe didn't run on every commit but maybe ran once every 30 minutes and just bundled any commits that happened within that? So maybe it's one commit, maybe it's four. Or would you have maybe preferred one where you didn't run the tests before deploying? You're just like, you know what? We trust that we ran them on the branch. It's good. We can just go straight to production.

STEPHANIE: Well, that idea, the second one of not running tests before deploying, I've never even thought about that. I think that it does provide some value because when you integrate changes into main, sometimes that might cause unexpected issues. So I think, in my experience, the times when CI failed, it usually was for a valid reason, and it wasn't just a blocker that we then had to retry the build for.

But what you were saying about bundling commits or a set of changes and then deploying on a scheduler maybe a few times a day automatically, that sounds really interesting to me. I have never worked on a team that has done it that way. But that sounds like it could be a good, happy medium between the two processes.

JOËL: I think that's effectively what the release manager is probably doing manually. But if there was a way to just do that automatically where you just say you merge to the main branch anytime you want, but on a timer every 30 minutes, the latest main will be run on CI, and if it's green, it will get promoted to production.

STEPHANIE: I think there's still a sense of whose job is it to follow up if something goes wrong in that sense.

JOËL: That's a good point. I think part of that is also it's a coping mechanism because of the slow test suite. We have a big process smell here, and we're trying to find some ways to get around it. And one way where I don't think any amount of process is going to help is when you have to do a hotfix.

So let's say there's a really bad bug in production. We need to get that fix out now, and so I make that fix. I live ping you to review it. We get it done in like 10 minutes, and we merge. And now we've got to wait 20 minutes for CI to pass before we can make that patch go live. And we're really hoping this test suite is not flaky because if not, we might be waiting another 20 minutes.

And so, in a sense, a slow test suite becomes a huge bottleneck to fixing emergency things. And now we're going to be tempted to say, "This is an emergency. We're going to bypass CI and just ship directly to production because this is on fire. We are corrupting customer data or something. This needs to be fixed now." And hopefully, we did not make the problem worse in our hotfix because we were rushing, which I have definitely done.

Luckily, in these situations, it has gotten caught by CI. But there have been situations where I've tried to do a quick hotfix that I thought was going to fix things, and then CI caught it, and I was like, I'm glad I didn't just put that directly in production.

STEPHANIE: Yeah, I think what I've come to realize is that the current process that I am experiencing on my client project, you know, I'm sure there's some history there about how it came to be and why they decided to do it that way. And that might be an artifact of something going wrong and them trying to put guardrails to prevent problems from showing up in production. So I do have some understanding there.

So if anyone out there has a deployment process that they love, I would love to hear about it. You can tweet us @_bikeshed or send us an email to let us know if you have a deployment process that works well for your team.

JOËL: Maybe we'll even feature it on a future episode.

STEPHANIE: Yeah, definitely.

JOËL: I'd like to get into some of the trade-offs that come with different processes, and one that jumps out at me from what you were talking about earlier is the impact of team size. With a smaller team, when you're, you know, 2,3,4,5 developers, you can have a really simple Git-based approach where merging a PR goes directly onto your main branch and maybe even have it set up to automatically deploy, and that's kind of it. If a commit is on main, it is live in production. And if you want to undo something, you just Git revert, and that goes live. And that's a really simple, effective workflow.

But then, as the team starts growing, you start needing something a little bit fancier because there are a lot of commits coming out at once. They might have dependencies on each other. Reverting becomes a little bit more complicated. As the product gets more complicated, too, then you start having to want to have work that's done, but you don't want to just have a PR sitting around waiting until go-live day. So I think that's definitely an axis to think of when you're thinking of trade-offs is some workflows work very well for smaller teams and others are a better fit for larger teams.

STEPHANIE: Absolutely. I think when you were talking about smaller teams, almost everyone has knowledge about what is currently being worked on. And so when problems do happen, that work of reverting or figuring out what went wrong isn't as hairy because most folks on a small team would know what changes are being merged and can pitch in to help there.

But yeah, I am really interested in the transition between a small team where you feel comfortable just merging the code and having the automatic deployment and when you do need to have a heavier-handed solution, I suppose. Do you think that there's an inflection point that pushes that decision to be made?

JOËL: I'm not sure exactly where that inflection point is. I might say as low as maybe 5 or 10 developers on your team, but there are probably some other variables that go along with that. Part of it might even be how good your team is at keeping commits small and focused, and independently deployable. If your team is committing really large commits that potentially break the build or that are tightly coupled to other commits, that might make it really difficult to say that your branch is always deployable. And so, you might want to bring in a heavier process earlier.

Whereas if your team is doing a lot of small, atomic commits, which I think we discussed this on last week's episode, I think that could probably allow you to get a lot more mileage out of a very simple workflow where even with a slightly larger team, you're still able to just merge and deploy and also potentially revert very easily because these are atomic commits.

STEPHANIE: Yeah. I like what you said about how you can get away with a lighter solution if you are really investing in things like making sure that each commit is green on CI. Because, you know, kind of what we were saying earlier, sometimes adding additional process without really figuring out what we're trying to solve here can lead to some of those trade-offs that we're talking about.

JOËL: Agreed. I'm a big fan of using the simplest process that your team can get away with. Maybe we could even extend that more generally to just use the simplest thing that your team can get away with. I think that goes for code complexity, that goes for maybe code optimization. Don't make it more complex just because you're hoping to have this massive scale one day because you don't need it today. So use a process that works for your team at your current team size, and then you can iterate on that and start adding more complex elements as the team starts growing.

So, Stephanie, I'm curious; we've talked about a lot of different types of deploy processes. What would be your ultimate favorite way to handle deploys if you had the choice?

STEPHANIE: I think I do prefer a more automated process. When I was on a medium-sized team, that was working pretty well for us. We were having deploys be kicked off when we merged to main, but then we had a Slack integration that would tell us, "Hey, your thing is being deployed." It would tell us the results of the CI build, and it would tag us if something went wrong. And so I think that was nice in solving that issue of ownership that I had mentioned where I knew that, oh, there was an issue. I have the most context, and I can solve it the most quickly on this team.

And then it was also good to just see what was going out, see what other people were working on. I liked that it made that very transparent. And that sense of feeling like you saw your feature from start to finish and seeing it live on production felt really good and gave me meaning in my development work.

JOËL: Yeah, that sounds like it hits a lot of really positive values, like you said, that ownership that you have from beginning to end, even with maybe the revert if something has to happen, the transparency where you get to see if any issues came through. And then the automation and the simplicity because it's just merge your PR and the work goes out.

Earlier in the episode, we were talking about trade-offs that come with a workflow. So a workflow like what you're describing, what size team do you think would be best suited for a workflow like that?

STEPHANIE: Yeah, I don't know if I have an exact number. I did mention that medium-sized team seemed to feel pretty good where we did have some investment in the infrastructure in place, so, like you were saying, we had guardrails when things went wrong. But it wasn't so much for a really large team where it would have been too noisy in the Slack channel.

And also, merge conflicts would come up if we were merging a lot of work during the day. And that did interrupt that queue and that flow and became something that we had to manually work through sometimes with other developers if that was their code that we had conflicts with. And so I can see it also start to not work past...I think I mentioned that the team was 20 to 30. I would be really curious to know how far that can take a team as it grows.

JOËL: So, 20 to 30 people, this workflow works pretty well. What about sort of maybe experience level? Do you think this is a workflow that requires a certain level of seniority? You're talking about merge conflicts a lot, so maybe a team that is very disciplined with keeping their commits small. Do you think that's required to make this workflow work well for the team?

STEPHANIE: That's a great question. When I was on this team, we did have people with all experience levels. And what I really liked was that it was okay if there were merge conflicts. It was okay if CI was red. People were super helpful in jumping on to work with you to figure it out, also, because they probably had things in the queue that they were waiting to try to go out.

But it felt like a team culture where we were all committed to releasing our code smoothly. And so sometimes merge conflicts would happen, but, like I said, you usually could see it and could jump in to help out if someone was maybe stressed out about it or needed an extra hand.

JOËL: I love the process you described. And the culture that your team had didn't require everyone to get it right all the time. There's room for mistakes or not even mistakes, but just less experience where you don't always know to scope everything super tightly, or your Git process isn't quite perfect every time. And that's great for a team because there's room to grow, room to bring in people of different levels of experience.

STEPHANIE: Yeah, I also think it's more realistic.

JOËL: Oh, 100%. I'd like to look at one more axis of trade-offs, and that is product type. What kind of product do you think that this workflow you described would fit well as opposed to maybe a different type of product that wouldn't be as good of a fit? I think what comes to mind for me immediately is maybe situations where you do a lot of work upfront, but then you only want it to go live for clients later, but you do want it merged. And so you decouple the Git history from actually releasing to customers.

So that's a product lifecycle that might be a little bit different. It could be a product where you even just do big releases at set intervals. So people don't want continuous change, but you're like, once every season, we release the new version or something like that.

STEPHANIE: Yeah, I was thinking that the continuous deployment process worked well for that team who was building a product that was very customer-facing in the sense that people were visiting the site every day. And they were running a lot of A/B tests on those customers as well. And so that was helpful because we could be releasing those tests iteratively and getting continuous feedback that way.

JOËL: So, as we discussed in this episode, no process is perfect. There are always trade-offs. So I think it was really fun to look at a concrete example of a process that you liked, Stephanie, and then look at maybe some of the trade-offs for when does it work and when does it not work so well? And with that, shall we wrap up?

STEPHANIE: Let's wrap up. Show notes for this episode can be found at bikeshed.fm.

JOËL: This show has been produced and edited by Mandy Moore.

STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show.

JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter.

STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email.

JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week.

ALL: Byeeeeeee!!!!!!!

ANNOUNCER: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

Support The Bike Shed