Three Parts: Change, Evidence, Sustained

How do we really know when improvement has happened inside a school or organization? In this episode, John Dues and Andrew Stotz unpack a clear, three-part definition of improvement and show why evidence, method, and sustained results matter far more than year-to-year comparisons. Their discussion offers a practical lens for leaders who want to distinguish true progress from noise and build changes that last.

TRANSCRIPT

0:00:02.4 Andrew Stotz: My name is Andrew Stotz and I'll be your host as we dive deeper into the teachings of Dr. W. Edwards Deming. Today I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. The topic for today is, How To Define Improvement. John, take it away.

0:00:23.3 John Dues: Hey, Andrew. It's good to be back. Yeah, I think this is really interesting. Apologies on the front end. I'm a little bit under the weather, so I may sound a little raspy today. But you know, one of the things that's really interesting is there's lots of claims of improvement. In my world, there's lots of claims of school im- improvement. I would even go as far as to say that those claims are like a dime a dozen, something like that. And the reason I say that is not to be mean or anything, but you know, I think that a lot of these claims, they're not grounded any kind of reasonable evidence. And I think sort of even beyond that, that claims are often made without a logical definition of improvement. So I thought in this episode we could talk about a three-part definition that makes it really easy to tell when improvement has occurred and just as importantly, when it hasn't.

0:01:21.9 Andrew Stotz: Exciting.

0:01:23.2 John Dues: Yeah. When I talk about this, I always like to start with a challenge. So, you know, if I'm in a workshop, I'll say, you know, get out a piece of paper and a pen so the listeners could do this as well and think about, you know, the successful improvement efforts that you've led throughout your career. So in my world, maybe it's increase in state test scores or maybe you improved student enrollment in your school. Maybe you did a better job at retaining the teachers in your school. It could be any number of things. Maybe it's decreasing student office referrals or decreasing chronic absenteeism rates in your school or your school system, which are two things on everybody's mind coming out of the pandemic especially. And I tell people, just create a list of those instances. And I give them a few minutes usually. And typically, people come up with eight, nine, 10 or so instances of improvement, whether that's teacher in their own classroom or principal in their school, or a superintendent thinking about the whole system. Then I say to them, now what I want you to do is pause and think, what does it mean to improve?

0:02:46.7 John Dues: What do you mean by that? And that really brings us to this important question. What is improvement? You know, and this was... Full disclosure, when I started thinking about this, I stumbled across the definition in a book I'll show you here in a second. But when I stumbled across this, you know, there was some conviction. And I think that probably a lot of educational leaders or just, you know, leaders in general would say, actually, I never really thought about that. I don't have an answer for this seemingly simple question. And like I said, I didn't have an answer to that question when I really thought about it, when I stumbled across the definition, probably for the majority of my career, maybe the first 20 years or so, if I'm at year 25. So, yeah, the first two decades, I would not have had a clear answer for that simple question. Now, I turn to this seminal work in the field of improvement science called The Improvement Guide. I'm sure a lot of people are familiar with this. And I'll share my screen so people can see the book and kind of share an interesting story about the book. And, you know, when you're... Can you see my screen all right now?

0:04:06.9 Andrew Stotz: Yep.

0:04:08.1 John Dues: So you can see, if you're just listening, you can see the covers of two books. So on the left, a lot of people will recognize The Improvement Guide. But there's an arrow up there. It says, second edition. And a lot of people will recognize that book. Probably less people. Maybe some people that have been doing improvement work for maybe three decades will know this other book, the first edition of The Improvement Guide. It's this purple book on the right, if you're watching. But there's this interesting anecdote that I actually think I might have heard maybe on your podcast when some of the authors were on. And almost as soon as they wrote this first edition, this purple edition, they got this note from this professor in Brazil, and it said, I know you guys are really big into improvement, and you're really big on operational definitions, but you've written this whole book on improvement, and nowhere in the book have you defined what you mean by improvement. So, you know, talk about a swing and a miss.

0:05:15.0 Andrew Stotz: Yeah. And just for listeners out there, you can go to the podcast.deming.org and you can search for quality as an organizational strategy with Cliff Norman and Dave Williams. We didn't talk about The Improvement Guide specifically, but definitely it's worth listening to those two.

0:05:33.3 John Dues: Yeah, and I think... So any of us that feel bad when we come to realize, like, how obvious that question is when we've made claims of improvement and don't have a definition. So even these guys that were writing a whole guide about how to do improvement missed as well. You know, and it's pretty obvious when you think about it that The Improvement Guide, that a book like that should have a clear definition for the central concept. That's right in the title. But it should be just as obvious to leaders that they also need a definition of improvement. And that definition should really precede any improvement claims then. So I thought it'd be interesting to take a look at the definition that the authors came up with in their improved second edition.

0:06:20.9 Andrew Stotz: Improvement in their improvement.

0:06:24.4 John Dues: Improvement in their improvement guide. Right. And the definition is really easy to follow. It's got three parts, and now I've adopted it into my own improvement work. But what they've said is improvement is "a change that alters how work is done or the makeup of a tool that produces visible positive differences relative to historical norms and relevant measures and it's sustained into the future." So we can kind of break that three parts of the definition down now. So in part one of the definition, what you have to be able to do is point to a change that was made that led to better results. You know, that could be a new tool you're using, a new approach, a new framework, maybe it's a new staff role, but something has to change, a new method, something has to change in what you're doing. So that's sort of part one of the definition. Part two is performance improved after the change compared to past results. So that also should be fairly obvious. So you did something different. You noted when you started this new thing, and at some point in relative proximity to when you tried that new thing, the data improved.

0:07:54.2 John Dues: You know, it went up. If that's the direction of good or, you know, if it's like chronic absenteeism or office referrals, you want it to go down. But you see that in the data after you've made this change, that's part two. And then part three is that improvement after the change was sustained into the future. So it wasn't just a temporary thing because you were paying a lot of attention to it, but you made the change. The data improved over time after you did this new thing, and then it kept going into the future.

0:08:27.9 Andrew Stotz: Which is the hardest part, by the way.

0:08:30.1 John Dues: The hardest part, I'd say too. Assuming you could bring about improvement, then sustaining it into the future, especially as you maybe take your eye off it a little bit, and then work on something else, that's very, very difficult.

0:08:43.1 Andrew Stotz: Yeah, the initial seizure that you get into of making change can be really powerful compared to the energy, you know, devoted to sustaining it.

0:08:55.9 John Dues: Yeah.

0:08:56.3 Andrew Stotz: And you could also argue, if something's not easily sustained, was it really an improvement?

0:09:02.8 John Dues: Right. I think that's just right. And, you know, what I do in the workshop then is then after I go over the definition, I ask people, now, think back to your list of successful improvement projects. You know, and for the listeners, if you pause, then you created your own list and then you heard that definition. Then I just asked the participants, would you revise your answer after reading this definition of improvement? And ask people, okay, now how many things, how many items are on that list? And a lot of people, if they're being honest, are left with none, actually. You know, because this definition sets a really high bar. But I think it is the right bar if you're actually interested in improving outcomes in your school, in my case, or in your organization. And I think what you often run up against is, this is kind of a simplified version of most improvement claims. But in my world, you hear claims, something like, you know, our state test scores improved. Right. The translation is, this school's or this year scores are higher than last year's scores.

0:10:25.8 John Dues: But that claim falls really short of that definition. Well, okay, the scores are better than last year. Well, what did you do to make them better? Also, a single data point is probably not enough to back that claim up. Let's instead turn to an example that meets the definition, and it'll help you understand how powerful this can be in practice. So, you know, let's suppose that you've been working to increase some student outcome measure. Let's say we gather it on a monthly basis, whatever this thing is. So now I'm going to show you a visual that has some data plotted over time. And the three parts of the definition of improvement have been labeled right on that chart. And having this visual is very, very powerful. This is when this definition really clicked. So I'll go ahead and share my screen again with you. All right. Can you see that chart?

0:11:35.9 Andrew Stotz: Yep.

0:11:36.9 John Dues: Okay, great. So first off, if you're watching that, you can just see very quickly that the claim of improvement has been substantiated with evidence. So for the people that are just listening, we have some data plotted over time for the first 15 months and it's kind of traveling along around an 80% average. And then all of a sudden that data shifts up substantially to now it's humming along at about a 91, 92% average. And it's much harder to sort of internalize this without being able to see it. But I'll do my best to explain it. So the first thing was, part one. There's a clear point in time where a change was introduced. So that's labeled as part one. Right in between month 15 and 16, there's an arrow there on the chart where some change was made. If this is a teacher's data, maybe they made some type of change to how they were instructing the class.

0:12:55.3 Andrew Stotz: In research, sometimes we'll call that time zero.

0:13:00.6 John Dues: Time zero, absolutely. And then second again, there's this clear difference in results after the change was introduced as compared to historical results. So that's part two. So there's this positive, visible difference relative to historical norms. Now, you know, this is an example for illustration purposes and I wanted to make it very clear, like this delineation, this definition. But in reality, you know, if you're a teacher trying this new method, for example, it might be that even if the method is successful, you're not maybe going to see the results immediately. Right. But this illustrates, what you're hoping to see. And it makes it very clear. And then part three on here, that improvement was sustained in the future. So you see the bump in scores in this, whatever this outcome is in this hypothetical in month 16. But it wasn't just months 16 and 17, it carried forth for another 15 months at this much higher level. So you can very quickly start to see there's this profound difference between most improvement claims and one supported by this three part definition. It makes it very simple. When do we introduce something new? And then what does the data look like over time after an initial baseline period?

0:14:36.9 Andrew Stotz: It reminds me of something I often say to people, which is, do you ever make the same mistake twice in your business? And of course, everybody says yes. And I ask them, imagine if you never made the same mistake twice, how would that change the outcome of your business? And then we have a discussion about that. But the point is that most people just live in a world where they never are able to really sustain improved performance. They just fall back to the same things. And this chart is a good way of understanding, have we truly sustained improved performance?

0:15:24.4 John Dues: Yeah, yeah. And you have to have a method. You know, that's why Deming would frequently say, you can't just have a goal. You have to be able to answer the question by what method? So that's why part one of this definition was calling out whatever change was introduced. Because in this system, in that baseline period from month one through 15, while there are some ups and downs in that data, it's really just bouncing around that 80% average. And if you don't change something in that system, then you're very likely just to keep getting those same results over time. Some fundamental change has to be introduced so you have a stable system, but it's not satisfactory. So you got to change something. And then you're going to keep gathering the data to see if that change had an impact. So again, it's not rocket science. And it was pretty intuitive to see that definition in that improvement guide. And then actually this sort of chart I'm showing you with a three part definition combined with a control chart or a process behavior chart, I saw this in their latest book called Quality as an Organizational Strategy.

0:16:42.0 John Dues: And when I saw the visualization of the improvement definition, which was only in like text form in The Improvement Guide, then it all just clicked. Oh, this is so obvious what this definition actually means and how you could tell if something has improved or not. So I think anybody that's doing improvement work, you know, whether you're in schools or some other type of organization, and whether you're the superintendent of the entire system, the principal of the building or the teacher in a classroom, all of you can use this. A student could use this, an athlete could use this. This definition, it doesn't really matter. It's sector agnostic and can be applied, you know, pretty widely against different contexts. But it makes it very clear how to tell when things are getting better, how to tell if, you know, maybe things are going the other direction or if they're just staying the same. Makes it very, very clear.

0:17:36.8 Andrew Stotz: Yeah, I mean, it's a great thought exercise that you started with that, you know, got me thinking and I'm sure for the listeners and the viewers got them thinking, like, what changed? Where have we really improved? And one of the hard things in business, and I'm sure it's the same in teaching, is that ultimately personalities and ultimately it's about the people, whether that's business or teaching. And one of the things that you can say about people that's commonly said, whether it's true or not, I'll leave you to think about that. And that is people don't change.

0:18:19.8 John Dues: People don't like to change.

0:18:21.1 Andrew Stotz: They don't like to change. And I would argue that they rarely change.

0:18:25.9 John Dues: Yeah, yeah, no, I would agree with that.

0:18:28.5 Andrew Stotz: I mean, the whole mission of life is to get to a point where you don't have to change. That is the human body, the human mind is just like trying to get to that point.

0:18:42.6 John Dues: Yeah. Unfortunately, it doesn't work.

0:18:45.6 Andrew Stotz: Yeah. And then you die.

0:18:50.1 John Dues: Yeah.

0:18:51.3 Andrew Stotz: But it just gets me thinking too, about... And for everybody, here we are at the... We're discussing this on the 9th of December. So we're getting near the end of the year, thinking about what we're trying to do next year and all that. And in my coffee business, as an example, we've had our shares of ups and downs, but we've tried to right the ship as far as making sure that we've got the right balance of profitability, the right number of staff, cash flow and a buildup of cash so that we have the resources to go after markets. And the question is, that I always have in my mind is, how do we prevent ourselves from slipping back into some old habits of maybe spending on marketing and sales and then not getting the delivery of that, and therefore the costs go up, but the revenues don't follow? How do we ensure that the improvements that we're making right now aren't just lost six months from now? And this starts to give me some ideas that I'm thinking about.

0:20:00.5 John Dues: Yeah. I mean, and once you get that improvement, like, how do you sustain it? How do you have the discipline to do that?

0:20:09.1 Andrew Stotz: Well, I think the first thing that this raises is, are we clearly measuring, first, whether the improvement happened?

0:20:19.3 John Dues: Yeah.

0:20:19.7 Andrew Stotz: And second, whether it was sustained?

0:20:22.8 John Dues: Yeah. And even before that, you know, just having a baseline. Most people have the data somewhere, but they haven't plotted it like this so it's clear what the typical performance is currently, a lot of people don't even take that step. You know, it's just last year, this year. Last month. This month.

0:20:39.8 Andrew Stotz: Yeah, that's good point. That's a good point. That's something you pointed out to me a long time ago about looking at, you know, my enrollments in Valuation Masterclass Boot Camp and saying, well, you need to understand, you know, what's the system you're operating in, and therefore, you've got to understand what that system can produce before you start thinking about, you know, what's your next steps.

0:21:01.6 John Dues: Yeah, yeah. And like I said in the middle of this episode, you know, this is a very high bar. This is not easy to accomplish. It takes discipline. It takes continual improvement. Dr. Deming talked about he liked using continual versus continuous because a lot of this would be discontinuous. You know, you have a focus, you may be improving an area, then you have to change your focus. But you can't keep your eye totally off this other thing. Like you were just saying, you have to kind of keep your eye on multiple things to keep an organization going. And that's part of the challenge of running an organization for sure. Be it a coffee business or a school.

0:21:39.9 Andrew Stotz: Yeah. It's interesting that we're talking about the definition of improvement long after we talked about the method of improvement, like PDSA.

0:21:50.5 John Dues: Yeah, yeah. And that's probably the best method I've come across. So, you know, in this visual that we're looking at, for people that can see it, you know, part one, a change was introduced. The most powerful tool that I found thus far is the PDSA cycle. That's where you would document the change. And again, this is for illustrative purposes, but in reality you probably have to run multiple cycles and kind of learn your way to a better system rather than you're probably not going to see this. Now, there are some things where you may see this start improvement between month 15 and month 16, but the reality is, in most situations that's going to take multiple rounds of PSAs and where there's sort of a gradual improvement over time. Again, you know, there are some things where you could see an improvement like this, but most stuff, it just takes sustained effort over time and continual learning and continual improvement type stuff.

0:22:51.4 Andrew Stotz: Yeah, I mean, this really helps me, and I'm sure for the listeners and the viewers to put PDSA in a very clear spot, which is, it's the tool for sustained improvement. Because you could imagine that in this chart where we have time zero and we have the past, let's say that's our starting point. We don't have any future data, but let's imagine that according to PDSA, we decided that we would try out and test out one particular method. And we find after testing it, no improvement, no change. Okay, that wasn't what we expected. Now we got to go back and adjust and then run the PDSA again. And then let's say we do that again and we find very little improvement. Okay, that wasn't what we expected. And our goal is to really try to get to a higher level of improvement. And let's say the third round, we get to a point where we get, oh, now this has sustained, you know this has produced improvement way beyond the others. The question is, can it be sustained? And that also has to do with the constraints of the system.

0:24:01.0 Andrew Stotz: Because if you don't have the proper, let's say, electricity, steady electricity supply, or you have problems with employees coming and going, bad training or whatever, you may find that you did make an initial improvement, but you weren't able to sustain it, so you couldn't really call it an improvement, it was more like a test.

0:24:24.0 John Dues: Yeah, yeah. These are all things that make improvement work so challenging, especially in a complex organization. Very, very challenging. No doubt. But I think this is probably a good place to wrap up and summarize. But I think just having this clear definition for the concept of improvement, I think there's sort of these three big ideas that I think from this episode that can put you on the right track. I think one is just recognizing now kind of being listening for this in your organization that most improvement claims lack evidence. So when you hear somebody in your organization make a claim, this went up, or this got better, or that got better, you know, ask for some evidence. How do you know? Let me see. Show me what you're seeing. How do you know this is improvement? So I'd call that sort of big idea one is, this idea that you need evidence.

0:25:27.7 Andrew Stotz: Yeah. The other thing I would take away from it, too, is actually in the process, you end up narrowing in on one thing. That chart is about one specific outcome.

0:25:42.6 John Dues: Yeah.

0:25:43.6 Andrew Stotz: And once you get so narrow to one specific outcome and you're tracking it and following it, it gets you more focused. And I think that to really get sustained improvement, you have to focus. And it's so easy to be distracted by the 15 things that need to be improved that you see as you walk down the hall in a school every day or as you... But in the end, true improvement is really hard. You can't improve 15 things. You can probably only improve one right now.

0:26:13.2 John Dues: Yeah, you gotta have a focus area for sure. What's the most important thing or the most important few things? Yeah. So the big idea one is, you know, from what I've seen, most of these improvement claims lack evidence. You need evidence. The second big idea is very simple. You know, you have to have this definition, and it's got to precede any improvement claims. Whatever your definition is, that this is the one I would use, this is the one I do use. But you know, before you can make a claim, you have to have a definition that clearly outlines when the thing has happened and when it hasn't. And in my mind, big idea three, is use this three part definition because it makes it so easy to tell when things have improved again and when things haven't. Yeah, after you apply these three big ideas, you know, I think you'll be able to answer the question, have we improved with conviction. You know, it makes it very, very straightforward. I like very straightforward things. And this is very straightforward.

0:27:14.8 Andrew Stotz: Yep. Most improvement claims lack evidence. John, on behalf of everyone at the Deming Institute, I want to thank you again for this discussion. And for listeners, remember to go to deming.org to continue your journey. You can find John's book, Win-Win W. Edwards Deming; the System of Profound Knowledge and the Science of Improving Schools on Amazon.com. This is your host, Andrew Stotz, and I'll leave you with my favorite quote from Dr. Deming, and that is that people are entitled to joy in work.