Balancing Ethics and Efficiency in Test Development

Stephanie has another debugging mystery to share. Earlier this year, Joël mentioned that he was experimenting with a bookmark manager to keep track of helpful and interesting articles. He's happy to report that it's working very well for him!

Together, they discuss tactics to ensure the easiest route also upholds app health and aids fellow developers. They explore streamlining test fixes over mere re-runs and how to motivate desired actions across teams and individuals.

---

• Raindrop.io
• Railway Oriented Programming

Transcript:

JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville.

STEPHANIE: And I'm Stephanie Minn. And together, we're here to share a bit of what we've learned along the way.

JOËL: So, Stephanie, what's new in your world?

STEPHANIE: So, I have another debugging mystery to share with the group. I was working on a bug fix and was trying to figure out what went wrong with some plain text that we're rendering from a controller with ERB. And I was looking at the ERB file, and I was like, great, like, I see the method in question that I need to go, you know, figure out why it's not returning what we think it's supposed to return. I went down to go check out that method. I read through it ran the tests.

Things were looking all fine and dandy, but I did know that the bug was specific to a particular, I guess, type of the class that the method was being called on, where this type was configured via a column in the database. You know, if it was set to true or not, then that signaled that this was a special thing. You know, I made sure that the test case for that specific type of object was returning what we thought it was supposed to. But strangely, the output in the plain text was different from what our method was returning.

And I was really confused for a while because I thought, surely, it must be the method that is the problem here [laughs]. But it turns out, in our controller, we were actually doing a side effect on that particular type of object if it were the case. So, after it was set to an instance variable, we called another method that essentially overwrote all of its associations and really changed the way that you would interact with that method, right? And that was the source of the bug is that we were expecting the associations to return what we, you know, thought it would, but this side effect was very subvertly changing that behavior.

JOËL: Would it be fair to call this a classic mutation bug?

STEPHANIE: Yeah. I didn't know there was a way to describe that, but that sounds exactly right. It was a classic mutation bug. And I think the assumption that I was making was that the controller code was, hopefully, just pretty straightforward, right? I was thinking, oh, it's just rendering plain text, so not [laughs] much stuff could really be happening in there, and that it must be the method in question that was causing the issues.

But, you know, once I had to revisit that assumption and took a look at the controller code, I was like, oh, that is clearly incorrect. And from there, I was able to spot some, you know, suspicious-looking lines that led me to that line that did the mutation and, ultimately, the answer to our mystery.

JOËL: Was the mutation happening directly in the controller? Or was this a situation where you're passing this object to a method somewhere, and that method, you know, in another object or some other file is doing the mutation on the record that you passed in?

STEPHANIE: Yeah, that's a great question. I think that could have been a very likely situation. But, in this particular case, it was a little more obvious just in the controller code, which was nice, right? Because then I didn't have to go digging into all of these other functions that may or may not be the ones that is doing the mutation. But the thing that was really interesting is that it seems like that method that does this mutating is pretty key to the type of object that we're working with, as in most places it's used, that mutation happens. And yet, it's kind of separated from the construction of that object.

And I was, I think, a little bit surprised that it wasn't super obvious that throughout the application, this is the way that we treat this special object. And I had wished that maybe that was a bit more cohesive or that it was kind of clear that this is how we use this in our domain. And it was, like, lacking that bit of clarity around how things are used in practice as opposed to trying to keep those in isolation.

JOËL: Yeah. Because now you have sort of two different diverging use cases for using this object that are incompatible, one that's trying to use the object as is and the other that's depending on this mutation. Now you can't have both.

STEPHANIE: Right. As far as I can tell, in most cases, you know, we're using it with a mutation, and maybe there is a good reason for those ideas to be separate. But it certainly did not make my life easier trying to solve this particular bug.

JOËL: I'm hearing you mention the idea of ideas being separate; definitely kind of triggers some pathways in my modeling brain, where I'm already thinking, oh, maybe this should be a decorator, or maybe this is just a straight-up transformation. We just have a completely different kind of object rather than mutating the underlying record.

STEPHANIE: Yeah, definitely. I think there could have been some alternative paths taken. At the time, that was kind of decided as the way forward for how to treat this particular domain object. So, that's my fun, little mystery where I got to, you know, play the detective role for a bit. Joël, what's new in your world?

JOËL: So, earlier this year, on an episode of The Bike Shed, I mentioned that I was experimenting with a bookmark manager to kind of keep track of articles that I find are helpful that I might want to reference later on. I wasn't sure if it was going to be worthwhile and mentioned that I'd report back. I'm happy to report that it is working very well for me.

So, the tool is Raindrop.io. They have an app. They have a website. And I have been sort of slowly filling it with some of my most commonly referenced articles. I had pulled some in initially, and then kind of over time, when I find myself referencing an article using my old-fashioned approach, which was just remember the keywords in the title and Google them, now I'll do that, link the article to someone else, and then add it to Raindrop so that the next time I'm starting to look for things, I have these resources available.

And I try to curate a little bit by doing things like tagging them and categorizing them so that when I need references for something, I don't just have to go through my personal memory and be like, oh yeah, what articles do I have on data modeling that I think might be a good fit here? Instead, I can just go to the data modeling section of Raindrop and be like, oh yeah, these five are, like, my favorites that I link to all the time.

STEPHANIE: Wow. We did a whole episode on how to search recently. And we totally forgot to mention things like bookmark managers or curating your own little catalog of go-to articles. In some ways, now you have to search within your bookmark manager [laughs].

JOËL: That's true. Sometimes, it's searching if I'm looking for a particular article, and sometimes, it's more browsing where I'm looking at a category, or a tag, or something like that, which maybe would have been another interesting distinction to explore on the How to Search episode.

I do want to give a shout-out to the most recent article that I looked up in my bookmark manager here, Railway-Oriented Programming. It's an article on how to deal with pieces of code that can error and how to sort of compose those sorts of methods. So, you now have a whole sort of chain of different functions that can error or not in different sorts of ways.

And it uses this really powerful metaphor of railway with different types of junctions and how you might try to, like, fit them all together so that everything connects nicely. And it's just a really beautiful metaphor. And I was doing some work on error handling, in particular, and I wanted to reference something and that was a great resource for that piece of work.

STEPHANIE: Very cool. I'm really intrigued. I love a good metaphor. I am curious: is this programming language and framework agnostic and not about Rails?

JOËL: Actually, so this is written on an F# blog. So, the code is all in F#. And it leans a little bit into some functional programming concepts, but the metaphor is more generic. So, it's a really fun way to think about when you're programming, and you're not just going through the happy path. But what are all the side branches that you might have to deal with, and how do those side branches come back into the flow of your program?

STEPHANIE: Very cool. I can also see some really excellent visuals here if you were to use this metaphor as a way of understanding complexity.

JOËL: Absolutely. In fact, this article has some pretty amazing visuals, so strong recommend. We'll link it in the show notes.

STEPHANIE: So, a few episodes ago, we talked about code ownership at scale because my client project that I'm working on is for a company with hundreds of developers. So, it's quite a big codebase, quite a big team. One of the main issues that I, at least, struggle with on a day to day is flaky tests in CI. When I, you know, I'm wanting to merge a change, I often have to run the test suite a few times to get to green and be able to merge and deploy.

And this is an interesting topic to me because when you're really trying to just get your changes through and mark that ticket as done, it's very tempting to just hit that, you know, retry button and let it sit and just hope for the best, as opposed to maybe investigate a little further about why that test was flaking and see if there's something that you could do about it.

So, I wanted to talk to you about the idea of making the right thing easy or how, at both a team level and an individual level, we can set ourselves and our team up for success rather than shoulder this burden [laughs] and just assume that things are the way they are.

JOËL: That's a really powerful question. Because I think by default, oftentimes, the less helpful thing is the path of least resistance, so, in this case, hitting that rerun button on a test suite, which I've absolutely done. But there's a lot of other situations in our work where, just sort of by default, the path of least resistance is the thing that's maybe less helpful for the team.

STEPHANIE: Ooh, I noticed that you kind of reframed what I said. I was using the term, you know, the right thing, but you then reworded it into the helpful thing. And that actually gets me thinking about these words are kind of subjective, right? What is helpful to someone could be different to what's helpful to someone else. And I'm kind of curious about your definition of the helpful thing.

JOËL: Yeah, I mean, sometimes it's very easy to sort of bring absolutes and [inaudible 11:26] judgments to code, you know, when we talk about writing good code, and being good programmers, being good at our jobs, not doing the bad things. And I think that sort of absolutism sometimes can, like, be very restrictive, and kind of takes us down paths that are not optimal for ourselves, for our teams, for our products. So, I'd like to think a little bit more relativistically have a little bit more of elasticity in the way that we formulate some of these ideas.

STEPHANIE: Yeah. I really like that reframing, and I appreciate the nuance there. I think for me, when I think of doing the helpful thing, I'm hoping to ease the day-to-day workflow for other developers because that also includes myself, right? Like, I've certainly been there feeling frustrated or just kind of tired of retrying [laughs] the test suite over and over again.

I'm also thinking about helpful, as in what will be helpful for future developers regarding the product? And can we make it robust now so that we're not dealing with bug reports later for things that we maybe we're trying to throw under the rug or just kind of glance over? Do you have any other guiding principles around what is helpful and what's not?

JOËL: I think that the time horizon you mentioned is really interesting because you have to balance sometimes short-term value versus kind of long-term. And is it worth it to maybe not fix that flaky test today so that we can ship as soon as possible? Or is it worth investing a little bit of time today so that tomorrow or next week is better? If you're a solo developer on a tiny project, that might be of personal benefit only. But on a larger team, you know, that might benefit not just you but a larger amount of people.

STEPHANIE: I just imagined the trolley problem [laughs] a little bit about, you know, the future developers and whose lives, not lives but whose happiness, the developer happiness you'll save [laughs] when you're on that track and making the decision about benefit now versus benefit later.

JOËL: No connection to railway-oriented programming, by the way.

STEPHANIE: I am really interested in also talking about barriers to doing the helpful thing or taking that extra step, right? Because I think identifying those barriers is really important to then, hopefully, break them down so that we are creating that path of least resistance.

JOËL: That's interesting that you mentioned these as barriers because I think, in my mind, I was thinking about the same idea but from, like, a completely mirror perspective, the idea of incentives. Why do incentives push you in one way versus the other?

STEPHANIE: I like that a lot. I think maybe there are, like, two different levers, right? Or maybe they are two sides of the same coin, where you do have incentive, and then you also have things that disincentivize you.

JOËL: This reminds me a little bit of the idea of the tragedy of the commons. So, in the case of flaky tests, everybody as a whole on your project has a worse experience. And project velocity slows when there are flaky tests. But you, as an individual developer, are incentivized to ship features quickly and efficiently. And the fastest way you can get your individual feature to production is by hitting that rerun button. Even though, collectively as a whole, every time we do that, instead of fixing the flakiness, we're adding a tiny, little bit of extra slowness that will accrete over time.

STEPHANIE: Yeah. It's kind of difficult to imagine really the negative impact that it's having collectively, right? You're kind of like, oh, I'm feeling this pain, but you're not always, like, really hearing about it from others, and we might just be silently suffering together [laughs]. I do think that once it's been identified as like, oh, like, we're all actually, like, really impacted by this, okay, great, let's make it a priority.

And so, now, let's say I am slightly incentivized to go and investigate the flakiness of a test rather than hit the rerun button this time around. I am wanting to talk about those barriers I was referring to a little bit because I've been in this position where I'm like, okay, like, I have some extra time today. So, why don't I look into this?

But then I go down that path, and now I'm looking at a test written by someone many years ago, you know, I don't know this person, and I don't know this domain. I don't know who to talk to to figure out even where to start. I may or may not feel equipped with the right tools to be able to address it.

And then, I think the biggest challenge is not feeling like it matters, right? Once I'm hitting this barrier, I'm like, is it worth the effort? At that point, maybe a little bit demoralized because, well, this is just one, and we have so many other flaky tests, like, what's one more?

JOËL: That's really interesting that you mentioned that sort of morale factor because it's absolutely the case on every flaky test suite I've seen. I think that kind of points to almost, like, an exponential cost to ignoring the problem. If someone fixed it early, yeah, it's slightly annoying, but you get it done. You fix it, and then you move on. When it feels like there's now this insurmountable pile of these and that any work you do here doesn't bring you any closer to the goal because it's effectively infinite, yeah, now, there is no incentive at all to do that work.

STEPHANIE: So, to avoid that, we talked about incentives, right? And I'm kind of curious: what ways have you seen or experienced that did make you feel motivated to take that extra step or at least try to avoid that point of thinking that nothing matters? [laughs]

JOËL: So, I'm going to start with, I think, what's maybe a classic developer answer, and I'm curious to get your thoughts on it because I think I have very mixed feelings about this. The idea of programmer discipline—we just need to kind of take more pride in our craft and pursue excellence, choose to do the right thing, even when it's hard every day. Because I hear a lot of that in our communities. How do you feel about that sort of maybe a bit of a mindset change? And how effective has that been?

STEPHANIE: Whoa, yeah, that's a really great point because I think I also feel quite conflicted about it. Because sometimes I can find it in myself to be, like, you know, I have the energy today to want to uphold, like, a certain level of quality that would make me feel good about doing my best work. And then, there are other days where I am, you know, just tired, [laughs] or feeling a little bit lazy, feeling just not confident that it will be worth it. Because there's also, I think, some external forces, right?

I've certainly been in the position where we were only rewarded or celebrated for shipping fast. That was the praise we were getting at retros. But then that actually really disincentivized me from wanting to do the helpful thing when the time came, right? When I'm in my development process, and I'm at, like, that crossroads. Because I'm like, well, I've been trying to do the right thing, but, like, no one is celebrating me for it. What if it takes away time from doing the thing that is considered successful? And, like, does it have an impact on me and my job?

So, you can, you know, kind of go down that spiral pretty quickly. And, in that case, like, no amount of personal, like, individual [laughs] feelings of responsibility can really overcome those consequences if, like, you're working on a team where that is just simply not valued, and there are other people who have authority to impart consequences, I suppose.

JOËL: Yeah. It's, you know, you have that maybe some amount of personal motivation. You feel like you're swimming upstream. And so, maybe the question then is, how do we reorient maybe some of the incentive structures in a team to try to make it so that if you are, let's just say, chasing some of those extrinsic rewards and you're trying to get praise from your teammates, or move forward in your career, whatever it is that is rewarded and valued on your team, how can that be harnessed to push people in a direction to get some of these extra tasks done?

STEPHANIE: Yeah. I will say, though, it is a little bit of both. And I think that was maybe why we're both kind of conflicted about it. Because on the individual level and, in general, knowing what my values are and, like, wanting to do good work and uphold quality, there are times where I don't always behave according to those values. Like I said, I am feeling tired, or maybe it's, like, almost 5:00 o'clock, and I just want to, you know, push my thing [laughs] so I can go and go on with my day.

What has helped me is having an accountability buddy. Maybe it's in code review, or maybe it's when I'm pairing, or maybe just talking about a problem that we're facing, right? And I might get into that sort of lizard-brain mentality of just wanting to do the easy thing. But as soon as someone else points it out or is, like, you know, like, "That's not quite aligning with what I know your, like, hopes and goals are for this project or how you want to do your work," usually, that's enough to be like, "Yeah, you're right." [laughs] And I'll take another pass at it.

JOËL: And I think we've kind of come back full circle here in that you mentioned that sort of the lizard brain side of you wants to just do the easy thing with the implication that, in this case, the easy thing is the thing that's maybe not useful to the team long term. What if we could restructure things a little bit so that the easy thing was the most beneficial thing for the team? And now you're not having to use discipline to fight the lizard brain side of you, but you're actually working with it.

One thing I've seen teams do with flaky tests is to not necessarily fix them immediately. Like, maybe you do rerun them, but when they happen, create a ticket for them and put them into the planning board. And so, now, these are things that get prioritized. They're things that might be fairly quick to do. So, it might be a fun, like, fast ticket for someone to pick up at the end of the week. It now counts towards velocity if tracked, so people, hopefully, get rewarded for doing that work. Is that something that you've seen? And how effective do you think that is in maybe making fixing flaky tests easier thing for a team?

STEPHANIE: Yeah, that is actually something that we are doing on my team right now, and I think it's great. I like the idea of tracking it, right? Because then you could also see over time, like, whether we have been getting better at reducing flaky tests, you know, then it's also really clear when someone is preoccupied with that work, and they don't get assigned something else.

One way that I've seen it not work as well as we hoped is when other work consistently gets prioritized over those flaky test tickets, and, you know, sometimes it happens. I think that's also information, though, about the team and how the team is spending its time compared to how the team thinks it should be ideally spending its time where, you know, we can say all we want, like, yes, like, we really want to make sure our test suite is robust. But when other things are consistently getting prioritized over it, then you can point to it and say, do we actually believe this if we're consistently not behaving according to that belief?

I found that challenging to have that conversation. But I do think that the concreteness of adding it to our workload for a given period of time is at least providing information rather than it being things that, like, developers are doing one-off or kind of just on their own time.

JOËL: Another solution that I've seen people do, and this is a classic developer solution, is tooling. If you have better tooling around a particular problem, sometimes you can shift that cost, that time of work it takes so that it's easier to do the best thing rather than to not...or at least make it easy enough that it's less of a big decision, like, oh, do I really want to invest that much work into something?

And a classic example of this, I think, is when you're trying to get into more of a test-driven development workflow. Having a test suite that's fast and, really importantly, having a near-instant way to run a test from your code editor makes a big difference in terms of adopting that workflow.

Because if the cost of running a test is too high, then yeah, the easy path is to just say, you know what? I'm not going to run a test. I'm going to just run it once at the end after I've written 100 lines of code or an entire feature. And now I am not getting the benefits of TDD. And that might kind of get into a negative cycle where because I'm not seeing the benefits, I do it even less. And then eventually, I'm just, like, you know what? Forget this whole thing.

STEPHANIE: I think that also applies to testing in general, where if it, you know, is feeling really challenging. I have definitely seen people start to get into that mindset of, is this worth my time to do at all? And it's a very slippery slope, I think.

That almost makes me think about, like, okay, like, what are some other ways to lift that task up and to elevate it into something that's worthy of saying, like, "Hey, like, that was really hard, and you did a great job. And that was really awesome that you persevered through that challenging thing." Sharing the pain points is really important, not only to, like I mentioned earlier, to, like, communicate that, oh, maybe, like, more people than you think are going through the same thing, but also to be able to identify when someone went out of their way to do the helpful thing and seeing that someone was willing to do it because, for them, being helpful is important.

When I see someone on my team take on kind of a difficult task to make things easier for others and then share about it, I feel really inspired because I think, wow, like, that could be me as well. You know, there's that saying that many hands make light work. And I also think that's true of tackling these kinds of barriers, where if we all feel this collective responsibility or, like, wanting to help out the group, it ends up literally being easier.

JOËL: I think something I'm hearing here as well is the value of giving praise. If somebody goes out of their way to make life easier for the rest of your team, give them a shout-out, whether that's in your team's Slack channel, maybe it's at an all-hands meeting, and you shout out some work that they've done or, you know, you put their name on a slide in your slide deck at some point, or whatever it is the mechanism within your team. Having a way to shout out people who've done some of this work that can be sometimes a little bit thankless is a great way to motivate seeing more of that.

STEPHANIE: Yeah. I was just thinking that it can be really powerful because, like you said, a lot of it is thankless, but also, we may not even realize the impact it's having on others until you give that shout-out and express, like, "Wow, like, that change, you know, I've been bothered by this issue for so long and, you know, that really made an impact on me," just keeping that cycle of gratitude going.

JOËL: So, I think we've kind of identified three maybe main areas of ways where you can help to incentivize these behaviors. You can do it through kind of process. We talked about the example of pulling the flaky tests as actual cards to be worked through on a board.

It can be technical by introducing some tooling that makes it much easier to do the work that you're trying to do.

And it can be personal by praising people, preferably in public, for taking that extra step. And I think all three of those can be part of a strategy to make it easier or more attractive for people to do work that benefits the team as a whole, even if they don't see an immediate return to that on a per-day level.

STEPHANIE: Yeah. I like that we kind of talked about these three different categories. And people in all different types of roles can, hopefully, take something from what we've shared, right? If you are a manager or leader of a team, maybe you can investigate your processes. If you're an individual contributor and you notice your colleague doing something that you, you know, kept meaning to but just didn't have time for, recognize that work. It really does take a holistic approach, but I think an impact can be made at every level.

JOËL: Agreed. I think for managers and more senior team members, that's really almost, like, part of their job description is to think about these kinds of things. How can we incentivize this work? How can we shift the team in a particular direction? There's a particular onus on them to do this right, to think about this, to model some of this behavior.

STEPHANIE: Yeah, absolutely.

JOËL: For someone who's really senior on a team also, they're often the ones who are tasked or who maybe take the initiative to build some of this more complex tooling so that these tasks are easier for more junior people. Maybe that's tinkering with some things and building an editor plugin that makes it easier to do some work. Maybe it's building a Rails generator so that the proper files get generated that maybe people wouldn't think to have when they're building certain work. Maybe it's building an RSpec matcher to make it easier for people to test some of the nuances of what we're hoping to do, catching some of these edge cases.

Whatever it is, sometimes there are things that the more junior members of our teams aren't aware of, and having a senior person take time out of their day to build these things so that now the entire team can be more productive can be a really helpful thing to do.

STEPHANIE: Yeah, that's a great point. And I think that also comes from having a pulse on what people are struggling with, right? So, you know, oh, it would be good to invest my energy into building a script to make this manual process easier because I keep hearing about people having issues with it or it being a challenge.

So, I would even recommend posing the question of, like, how do people feel about being able to fix that flaky test, right? Like, is it intimidating? What are those barriers? Because your team knows best about what that experience is like. And if that is not something on your radar, maybe there are opportunities to incorporate it into where you're evaluating team morale and happiness.

On that note, shall we wrap up?

JOËL: Let's wrap up.

STEPHANIE: Show notes for this episode can be found at bikeshed.fm.

JOËL: This show has been produced and edited by Mandy Moore.

STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show.

JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter.

STEPHANIE: Or reach both of us at hosts@bikeshed.fm via email.

JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week.

ALL: Byeeeeeeee!!!!!!!

ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com.

Support The Bike Shed