Clayton Lengel‑Zigich: Welcome to another episode of the “Agile Weekly Podcast”. I’m Clayton Lengel‑Zigich.
Jade Meskill: I’m Jade Meskill.
Derek Neighbors: I’m Derek Neighbors.
Roy vandeWater: And I’m Roy vandeWater.
Soul Crushing Technical Debt, Where To Start
Clayton: And today we are talking about Soul‑Crushing Technical Debt.
[groan of disgust]
Clayton: [laughs] Yes. Times a thousand.
[laughter]
Clayton: Roy, you coined that phrase, moments ago. Can you explain that a little bit more?
Roy: I’m just thinking of the technical debt where it feels like it’s so intimidating that you don’t even know where to begin. Every effort feels like you’re blowing into the wind, or trying to push a mountain with your pinky toe, or something.
Clayton: Yes. I was looking at some code with someone today, pairing, and trying to work through some stuff. There was some code where no one really knew how this thing worked. If you changed the seventh constructor argument, there were 10 total, if you changed the seventh constructor argument from an empty string to null, then it did something totally different, and it blew up everywhere.
I kind of was feeling the same way. There’s this core part of the application that nobody understands how to use it, and unless you’re sitting there with Eclipse open where you can actually look, and it will tell you all the different crazy details, you would have no idea how anything works.
[laughter]
Derek: That’s OK. I’m winning [inaudible 00:01:21] grudging technical [inaudible 00:01:23] GOTO statement in Java, so there.
[laughter]
Clayton: We didn’t see that.
Makes You Want To Do The Re-Write
Roy: That was pretty intense, but it makes you want to do the big rewrite, but some of these applications are so gigantic. First off, the big rewrite is always a mistake. I’ve never seen that go well. But, even a tiny, it feels so big that I don’t know where to start. I feel like anything I do just does nothing.
Clayton: I think also this is when you get into the habit, or where people mistakenly call it refactoring.
[laughter]
Clayton: They say, “Oh, we just [crosstalk] need to refactor that,” which is “We need to rewrite that.” Especially even if they have a test suite, I don’t think you’re refactoring. Even if you have a test suite, if you just go through and just rewrite a big swath of the code.
Derek: We’d have to rewrite every test. I refactored it and now it works better!
Roy: Red, green, wait a decade, refactor.
[laughter]
Roy: Is that how it works?
Reasonable Strategy Out Of Soul Crushing Technical Debt
Clayton: I guess the thing I’m concerned, or I’m curious about, is it a reasonable strategy to say we don’t want to do the rewrite, but we know that there’s parts of this application that we don’t like. Can I take Boy Scout rule and when I’m in some part of the code just try and make it a little bit better?
If, for instance I’m trying to use some PC application that doesn’t really work very well, can I just make some new endpoint or some new interface. And I can just use that thing and I can slowly piece it together. And if I do that over and over and at the end of the day, I would have this natural thing that one can use. But it doesn’t feel like a rewrite.
Derek: No! Even that feel like is marginally better. I feel like it will take years and years before even, be able to notice the changes.
Jade: Sometimes it does, right.
Roy: That’s the author of a lot of technical debt.
Derek: That is so crash‑y.
Your Mindset Has To Change
Roy: I can say that, we have to take that strategy before, or we make small incremental improvements as we go. What I tried to do in later days as I realized how bad things were is do the strategic rewrite. Rewrite small pieces at a time, try to make them a Modular. It still have to fit into the overall mess or nightmare. But those pieces will be nice and clean.
It was really more of a mindset than necessarily a whole bunch of Codes that we had to rewrite. We have to start thinking about things in different ways in order to overcome not so crushing technical debt. We have to get a whole new religion to get over our depression from what we have before.
Jade: Such really tough too. I can hear a whole bunch of people never seen any Code other than this, one project. They don’t any other way, they don’t even understand that there is a better way.
Extracting The Monolith
Derek: To me, is really most technical that people have created monsters. And they keep adding more, and adding more, and adding more. And I think what Jade might be talking a little bit more about is when you start to take a more unique approach instead of saying, let’s have a big monolithic Operating System that does just everything.
We have this things that have series of tools that we can change the tools altogether. We can [inaudible 00:04:23] and we can do so really complicated things but by using some very really simple things.
A lot of time, you have to take these big giant monolithic applications that nobody really understands from cradle of the grave and start to say, “Can we take a little bitty piece of it, this little component out of it?” And we pull that out and make it an API or a Library or something that is more independent and everybody can understand what that thing does. You kind of do that one at a time, one at time, until pretty soon, you’ve now got several smaller applications or some applications or some Libraries. So different things.
But now it’s much more testable, much more readable, much less interdependent on everything. That’s where technical debt kills, where everybody is afraid to touch it because they have no idea what actually been impacted by.
Fear Paralyzes And Creates Inaction
Roy: Like we had that GOTO today that we were arguing about like why even sit and make fun of it, why don’t we just fix it. It’s like, everybody is too scare to touch because who knows what the hell they were thinking when they did that.
Clayton: Well and that’s like the big thing that Bob Martin goes over is, you get code rot and you get all these bad practices because people are afraid. That’s like the fear of changing that code is probably the worst that you can have in your [inaudible 00:05:32] .
Jade: I agree.
Derek: Well and I think that is the smell of technical debt, right.
Clayton: Right.
Derek: The minute that you are afraid to change your code; you probably have some form of technical debt. The more afraid you are, the more soul sucking, soul crashing that, that has become.
Self-Awareness About The Smells Of Technical Debt
Clayton: How much technical debt or maybe how bad does it have to be or do you think before the average team is realizing that that’s inhibiting them from either being more successful or having more “progress” and how you will you measure that? Maybe shipping more features or being able to deploy more frequently or having a 10 minute [inaudible 00:06:07] or whatever it is?
Roy: I don’t know but a month ago my opinion would have been way different than today.
Clayton: I think it’s right about the time you got a business.
[laughter]
Clayton: That’s probably realistic.
Roy: I’m willing the team whatever realize it on their own because they’ve been boiled into it slowly.
Derek: The only time I ever see teams come to that realization is if they have somebody new added to the team that says, “Oh my goodness, what are you doing?” Or if they get to a point where they keep adding crap on and somebody asks…You think of a Christmas tree and it goes up to a point and then somebody says, “I’ve got a 7,000 pound star I want to put it on top of the tree,” and they go “Ugh! We need to rewrite this so that it can support a seven…”
And that’s usually what it comes down to is, “Yes, we can do the 7000 pound star but we’re going to need three years to rewrite the application to accommodate that.” If it really is a 7000 pound star very often the business says, “It’s worth a whole lot of money, so I guess we’re going to give you three years to rewrite it.”
Clayton: And then it doesn’t support the star and it takes six years.
Derek: Right.
Jade: [inaudible 00:07:13] deal with a 100 pound star.
Clayton: Well and then we just build up a whole other set of technical debt, right.
Derek: Correct, because we didn’t learn anything from [crosstalk] . We never had to deal with fixing the problem; we just got to start it from scratch.
Achieving Technical Debt Englightenment
Clayton: How do you get enlightenment?
[silence]
Jade: That’s a tough question.
Clayton: I was just waiting for that to be totally silent there for a second.
Jade: [laughs] Good thing we have all the answers on this podcast.
Derek: I know.
Clayton: I think some it…
Roy: I needed the answer out.
[laughter]
Jade: It’s a secret.
Derek: For a mere $3000 dollars for a minute we can tell you things…
Jade: Get out of technical debt.
Being Ready To Hear The Truth
Clayton: I think there’s two kind of elements to it, one is I think that people have to be ready to hear it, have to be ready recognize it. Sometimes I think people are in kind of that mind set or that thing of, “Who are you to tell me that I’m potentially doing something wrong.” If I’m not ready to be there it doesn’t matter.
Roy: It’s a 12 step program?
Derek: You have to be able to make you have a problem.
Roy: There’s a higher power and the higher power has control over the code.
Derek: I think the other thing is until you kind of get a little promiscuous in other code bases, I don’t think you understand the deficiencies that you have. I know from myself the big thing we’re starting to participate in open source and free software, that really gave me an introduction to a whole bunch of developers working with them and hearing their opinions and hearing how they thought this was damn or that was great or this was…
Clayton: They reject your patch?
Derek: They reject your patch. I think that that opened up a whole new world outside of the limited scope. Even though I might have been in a company that had 100 other programmers, there’s a bunch of group think within that group of 100 programmers.
The minute I started to step out and deal with different platforms and different countries and different ages and all sorts of different demographics about they viewed things, it started to open me up to like, “Wow, maybe I’m doing some things that are really stupid, maybe I’m doing some great things, maybe I’m doing some bad things.” It was the first time that I actually reflected and said, “Wow, maybe I should look at what I’m doing.”
Jade: I had a similar more experience, I came mostly from a self taught, but a kind of isolated path right to development and once I really started getting into supporting free software and kind of drinking the kool aid there, that really opened my eyes to another whole other world of doing things and really an intolerance for crappy code.
Roy: I think pairing helps a ton with that too, but even with pairing somebody in the group of people that are pairing need to know some different practices.
Jade: If we both don’t care…
Roy: Right, we both don’t care or we both…even if we care but we both only ever worked on the same code base and we don’t know any other way like, “We’re never going to be able to teach other good things.”
Derek: I think a lot of it is getting out the same technology. If I program in .NET all the time and I have no concept of what UNIX is, there are some huge paradigm differences between those two operating systems and platforms. There are good things and bad things about both of them, but if I only know one of them, I am totally ignorant of the other.
Same with languages. If I’ve never used a dynamic language or a statically typed language or a compiled language aversion, interpreted language, I have no idea how the other half lives. I think you can take so many things back, even if you just go play around with the project or with some code for a couple of days and figure out how some things work.
You can take that experience back to what you’re currently doing and I think that is the thing that is really difficult. Until you’re doing that, I think you have no self‑awareness. I think you have to actively seek to be self‑aware about how you write software.
How To Pay Down Technical Debt
Clayton: So how do you go about paying off the debt? Is that something that you do?
Derek: Add more people.
[laughter]
Clayton: Assuming you added infinity people…
Roy: [laughs]
Clayton: …and your project is good.
Derek: I would rewrite it then, because now you’ve got lots of people to rewrite it.
Jade: Skunk or a kitten?
Incremental Approach
Clayton: Do you try and just take the incremental approach and do it? Maybe you inflate everything, like every new feature you inflate because you’re going to try and tackle with the debt. You kind of, “Hush, hush.” You don’t tell anybody you’re doing it, but you do it anyway.
Jade: How do you pay down normal debt? No, I’m really asking. I need to know.
[laughter]
On A Team, Your Debt Is My Debt
Clayton: You don’t normally share a normal debt amongst a bunch of people, like you share it amongst your family.
Roy: That’s why my wife [inaudible 00:11:46] .
Clayton: But, with a team, many teams don’t even see themselves as that much of a team, so they don’t even think of the debt as a team ownership thing. They think of like their own specific screw‑ups a maybe a small part of their debt, but they don’t own the code base because, “I didn’t touch your part of it. That’s your problem.”
Roy: I think that’s a good point. I don’t think you want to lie about it because what’s going to happen is I’m just going to end up trading all sorts of other problems and team dynamic issues and trust issues and everything else. But if you were to look at normal debt, you would say, “The first thing we have to know is, “How bad is it? How much do we owe?”
Then, you have to know, “How much do we make?” Then, you have to know, “What’s an acceptable timeline to pay it back?” Based on that you’re going to have to make all sorts of decisions. So if you say, “I’m X‑thousands of dollars in debt. I make X‑amount and it needs to be paid in the next 12 months.” I know how much I need to cut my spending or increase my revenue in order to pay that in that time.
Derek: What happens if you can’t? What happens if your income is not enough to cover the debt in that timeline?
Clayton: If you’re listening to this podcast and you have a manager who treats it like, “Man, if you have technical debt, you screwed that up. That’s your bad. You’re going to have to figure that out on your own because you still need to make up. You need to deliver all these features. You need to have 20 velocity.”
End Of Life The Product
Derek: That’s what I put on my resume. At some point, you have to be realistic. There are some projects that are so far in the technical debt that it would probably be beneficial for people to say, “We’re going to ride this product out to its death and we’re not going to add more code to it.”
Jade: Adjust the partner.
Derek: We call it end‑of‑life in a product for a reason. There are times where a product has run its course. Meaning if the product is not able to boost revenue enough to pay to remove the technical debt, it’s probably not a product worth continuing to operate.
Roy: That’s a good point.
Jade: Sometimes the cost benefit ratio is so out‑sized because it takes so much effort to add the smallest feature because of all of this technical debt.
Roy: Right.
How Hard Is It To Add A Feature
Jade: I think that becomes your compelling selling point. If you’re trying to maybe turn the corner before it’s too late. You really need to look at and measure, “How hard is it to add a feature? How many defects get generated every time we add a new feature because of intended consequences? How do we start to mitigate those things?” But you need some metrics around that.
Clayton: I think it’s really difficult to measure how the technical debt and the coupling of the code and all that other stuff impacts the feature adding. I think that’s a difficult skill to acquire and a lot of teams really don’t have that.
Derek: It’s not teams, it’s poor product ownership. This is a classic case of I’ve got a product that’s making $10 million a year and it’s been making the same $10 million a year for the last three years, i.e. it’s got no growth. But, I’ve got a team of 10 people that I’m paying to add new features to it.
If I’m paying three teams worth of people to add features and I’m not getting any additional revenue, I have to ask myself, “Why am I not having two or three people maintain this program and let it ride itself out for the next two, three years making $10 million, take those other three teams and go build something new. Not rewrite this out again, but go build something new that could be a potential new revenue generator.” I think people are just afraid to admit those kinds of things.
Clayton: I think we are out of time. I hope we have adequately addressed the soul crushing technical debt.
Roy: My soul feels kind of crushed and under pressure.
[background noise]
Clayton: We’ll leave like 11 hours of silence on the podcast.
[laughter]
Clayton: Somewhere in there, we’ll answer the question about how you solve this problem.
Roy: We’ll come back in with it.
Clayton: So keep listening.
[laughter]
Derek: I’ll end at 17.
Clayton: Thank you.
[silence]
[background noise]
Clayton: My soul still feels kind of crushed.
Recording: The answer is 42.