Understanding system variation

Are you looking for smarter, more sustainable improvements? In this episode, John Dues joins Andrew Stotz to challenge the way schools - and businesses - set goals. Instead of chasing stretch targets that often frustrate teachers and students alike, John shows how to use data and systems thinking to create real improvement.

TRANSCRIPT

0:00:02.2 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we dive deeper into the teachings of Dr. W. Edwards Deming. Today, I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. The topic for today is how to set educational goals. Take it away, John.

0:00:25.6 John Dues: Hey, Andrew. It's good to be back. Yeah. So the thing that I see just over and over again is that goal setting in schools is just really ineffective. And not only that, but it's often counterproductive. So I thought it'd be good to talk about a different approach to how to set educational goals. I think most typically what I see, and this is probably not just true in schools, but in other places too, you get some result, and then you set a goal to increase by 10 or 15 or 20%. But I think that what often is missed is that without certain conditions in place, these stretch goals end up disconnected from reality. And so what I wanted to do in this session is to teach three conditions that I believe will lead to much more effective goal setting.

0:01:20.8 Andrew Stotz: It's interesting because goal setting seems really simple, like, just set a goal and go get it.

0:01:29.1 John Dues: Yeah.

0:01:29.5 Andrew Stotz: And yet when you're... You say that it's simple as an individual that I'm going to go walking every morning. That's my goal. It seems simple as an individual, but when it gets to a company level, it can get really complex, or a school level. And also there's this... What did you call it? Like aspirational... Or what was the word? Stretch, stretch goal. It's so common, particularly when business is not going great. Like, I want this from this team, and it's just so common to say, to lay down demands. I want 20% revenue growth. I want students passing this by 70% or whatever. And yet it just doesn't work that way. And so I'm looking forward to hearing from you about your three conditions. So let's go.

0:02:23.1 John Dues: Yeah, sounds good. So let's start by looking at why I don't think stretch goals are helpful. And since I'm working in schools, I like using data that people are familiar with. So I'll use this third grade reading state test score data. It's very common. It's really important. There's often some guarantee around third grade reading scores or there's often high stakes attached to third grade reading. So I'll use that. And so let's say in year one, and this is actually real data, 54.9% of third graders scored proficient or higher on that test. And then in the next year, they score 63.8% of the third graders scored proficient or higher. So in most places, leaders and teachers would celebrate because that represents an 8.9% increase in proficiency rates. And so then they do something like set a stretch goal of, well, next year we're going to grow by 20%. So you think, well, we grew by nearly 9%, so we can surely hit 20%. But then what often happens in a situation like this is that the next year's data comes out and actually goes down.

0:03:41.4 John Dues: And that's exactly what happened in year three of this data. Now, in this third year, only 61.2% of the students scored proficient or higher. And so then you think to yourself, well, what stretch goal would I set next? Would you change that 20% goal because of the decreased rates in the most recent year? So it actually is an interesting exercise. If you're listening to this, actually get a number in your head. You're the school leader, you're the superintendent, you're the principal or whatever, think about what goal then would you set for this group. So they grew by almost 9% and they went down by a couple percent. Between year one and two, they set the goal of 20%. Now what are you going to do? What's the goal? What are you going to say?

0:04:36.2 Andrew Stotz: And I'm thinking about the tampering concept that Dr. Deming talks about, constantly adjusting based upon where you're at. And it's interesting because in that case, you may say, you know, a disappointing year, but we've got to stick to our 20%. We got to stick to our strong, we can't give up. Remember that celebration we had?

0:05:01.0 John Dues: Yeah, this is a one year blip. We had increased by nearly 10. I know we can do it if we just double down. Let's keep that 20% goal in place. Yeah, that's definitely what a lot of people would do. So let me show you the results for those three years and then what happened for the next several years after that. So I'll share my screen for those that are watching so they can see this actual data in chart form.

0:05:27.7 Andrew Stotz: Great. Yep, we can see that.

0:05:29.8 John Dues: You can see it. All right. So now we have nine years of test data, and we can see that setting that early stretch goal had no effect whatsoever on outcomes. And so in year one, we had that 54.9% of those third graders were proficient. Then we went up to 63.8 like we said. Then we dipped back down in year three to 61.2, and then in the subsequent years that we're looking at now, it jumped up to 60.7, and then it fell to 51.9, and then it jumped up to 59.8, and then it rose again to 62.3, and then it rose again to 64.5, and then it fell again to 61.3. So again, between year one and two is when we set that stretch goal of 20%, and we can see again that it had no effect on outcome. So really, in all likelihood, the goal was not only ineffective back there between year one and two, but it was likely counterproductive. Because if you're thinking about, if you're a teacher in this school, you're teaching third grade or you're the principal of the school, year after year, this team didn't come close to approaching that 20% stretch goal that had been set.

0:06:52.8 John Dues: Year one, year two, year three, all the way through year nine. Some years the scores go up, some years the scores go down. It's pretty demoralizing if you think about it. I think the good news is, and the reason we're having this conversation is I think there's a better method for understanding your data and then using that data to set school goals in a smarter way. So I'll introduce these three conditions. I think about them as three conditions of healthy goal setting. So the first thing that you want to do is gather baseline data in the area that you're interested in improving. So this could be historical results if you have data, or it might be new data that you need to collect if this is a new focus area. But either way, you need some type of baseline as a starting point, and then you can work to understand these three conditions.

0:07:55.9 John Dues: So the first condition, condition one is what I call the capability of the system under study. And by system under study, in this case, I just mean the third grade reading test data. So how capable is it? Condition two is the variation of that system under study. And then condition three is the stability of that system under study. So those are the three conditions that I'm interested in taking a look at. And this focus could be on any data that occurs across time. It could be like state test scores, like we're looking at here. Could be attendance rates, could be oral reading fluency rates for those elementary teachers. It could be the number of office referrals that are coming into the principal's office or assistant principal's office on a daily or weekly basis. For this example, I'll just keep using the third grade state testing data.

0:08:52.8 Andrew Stotz: And let's just go through that just to review. So you talked about gathering baseline data. So you got to figure out what's happening here. The second thing, the first thing then you talked about, step one is capability of the system under study. And what does the word capability mean?

0:09:10.5 John Dues: Yeah, we'll get into each of them and then define each one.

0:09:12.0 Andrew Stotz: And then the second one is the variation of that system. Okay. Now what I noticed in all of these as you're talking about system, third one is stability of the system. So I guess a big part of this is basically saying if you're not looking at it as a system, you're not going to have the success that you want to have. So that's certainly a reinforced point there. Okay, keep going.

0:09:38.1 John Dues: Yeah. And by system, I just mean there's all these inputs into the third grade reading test, and then there's things that we do in schools, and then there's an outcome, there's an output that happens. So that's why I'm calling this third grade reading state testing data system, basically. So the first thing that I've done is taken that data that we've been looking at in a table and I put it into what I call a process behavior chart. Other people call a control chart or Shewhart chart. We've talked about those charts here, and I've done that so that we can better analyze those three conditions. So this charting method allows us to do that. So can you see the chart now?

0:10:22.6 Andrew Stotz: Yep. And for the audience, we have a chart up on the screen, third grade English language arts test, state testing proficiency rates. And maybe you want to describe what you got in the chart.

0:10:36.1 John Dues: Yeah, so it's the same data that was in the table, but now I've just taken it and plotted it. And that's what the blue dots are. So for each year, which is running along the bottom of the chart of the x-axis, there's a year, year one through nine. Those all represent school years. And on the y-axis I just have proficiency rates running from 0 to 100%. And then I've plotted that data that we had just looked at for each of those years. So it's the same exact data. So the thing that should... Before you even get into the chart in detail, the thing that should jump off the page is how much more intuitive the analysis is just by looking at the chart as compared to looking at that same data in the table.

0:11:23.8 Andrew Stotz: Yeah, in some ways it just looks like a boring chart that looks pretty normal, a little ups and downs.

0:11:30.7 John Dues: It does, but it's super powerful. And it allows us also to start looking at those three conditions that we just talked about by putting it in this format, taking that data from the table and putting it in the process behavior chart format. So you had asked about capability. So what I would say is that the capability of this third grade ELA or reading system is actually defined by the chart. So some of the most important components of this are that green line running through the middle for those watching the video is the average of the nine data points that we have. So the average of those nine tests is 60.7%. That's the green central line. And then those red lines, there's the bottom line and the top line. Those are called process limits. And they're set by plugging nine years of data into a formula.

0:12:28.8 John Dues: So the most important thing is I don't get to choose where those limits are placed. They sort of appear on either side of the nine data points. So that lower limit is a little bit lower than the lowest point. So it's set at 44.9%. And the upper limit is a little bit higher than the highest data point, and it's set at 76.5%. This data, or this really tells us that the data is really just likely to bounce around this 60.7% average, but could plausibly range between those two limits.

0:13:09.8 Andrew Stotz: So is that when we say capability of the system is that this system, as is could just by randomness, could end up between 45, is highly likely to end up between 45 and 75 without it necessarily being attributed to anything but random variation.

0:13:31.9 John Dues: That's right, yeah.

0:13:33.0 Andrew Stotz: Yep.

0:13:33.6 John Dues: So another way to think about it is given what we're seeing in this nine years, if we call this our baseline, the data is likely to bounce around that average of 60.7%, but it could plausibly range between those limits.

0:13:51.9 Andrew Stotz: And one of the fun things I was just thinking about is if you made the average and the blue line of the data like invisible, then you'd have a really good picture of the system, right?

0:14:05.6 John Dues: Yeah, that's right.

0:14:06.3 Andrew Stotz: The capability is between these two fixed lines. There it is. There's your capability. Okay.

0:14:13.3 John Dues: That's the capability. And the thing that we can tell for sure, well, that's fairly certain, with a high degree of probability, we can tell that this third grade reading system as it's currently set up is incapable of hitting that 20% stretch goal. And the reason that I know this is if I add 20 to that average of 60.7%, the answer is beyond the upper limit of 76.5%. So is it possible that you could hit that stretch goal? I would say it's not impossible. It's highly, highly unlikely. And nowhere in that time period did we get anywhere close to hitting that 20% target. And so when you start to look at data this way, it becomes self-evident that the 20% stretch goal in this case is...

0:15:16.5 Andrew Stotz: So then a crafty manager would say, okay, John, that's great, so I'll set the stretch goal at 75%. We should be at the upper limit and therefore just below the upper limit. It's within the capability of the system. Let's go.

0:15:33.5 John Dues: Yeah. So that target setting is beyond the content of this podcast, but to me it would depend on what's the nature of that target. Is it an accountability target that a manager has set for subordinates and that people are going to be held accountable in some way? Demotions, lose their job, affects pay, that type of thing. That's completely the wrong way to go about target setting. But if the target setting is, hey, look, it's a fact of life that more kids need to be able to read proficiently. We need to, as a team, figure out how we're going to bump this up every year until we get to 80, 90, 95, 99, 100%, that type of thing. That's a completely different sort of mindset. That's an improvement mindset, and I would be all for it even if the target is outside of the capability. I'd be all for it if there's an improvement orientation and team orientation to that target setting. So that's the first condition, that's the capability of the system. So the second condition that we'll look at here is that we want to understand the variation in the system. And so the data now in this process behavior chart format can help us understand the variation in this system.

0:16:55.6 John Dues: So the first thing to notice when it comes to variation, you want to look at the blue data points and how they orient themselves around that green central line. And so you'll notice if you're able to see it, that those blue data points are really just regularly bouncing above and below the green line. There's nine of them. And if you look closely, three of the points are below the line, one point is just about right on the line, and five points are just above that average line. So if you're doing year to year, the results increase and then decrease, then increase, then decrease, then increase slightly for three points in a row before decreasing in that most recent year.

0:17:50.2 John Dues: So again, given the data we have so far, it's clear that this system is not going to increase from that average of 60% or so to 80% proficiency rates in any given year. It's just not going to happen based on what we're seeing in this baseline data. So again, the table, it's much harder to see. I mean, you can see it if you're really looking, but it's much less intuitive when you don't have the plots or the dots plotted and then connect them by line so you can see the ups and downs. And the other problem is, so often with something like state test data, but with lots of data, we only have two or three of those points in a table. And we're just looking at did we increase or decrease from last year.

0:18:41.3 John Dues: And so we completely miss all of these patterns that are happening in a relatively recent time period. So by charting, we get that analysis that jumps off the page when you look at data in this way.

0:18:55.7 Andrew Stotz: Yeah you feel like when you get a table of two data points, you get an understanding of data, but when you look at it like this, you get an understanding of the system.

0:19:09.0 John Dues: Right. And if someone's hearing this for the first time and you hear, okay, system capability, system variation, and the third condition, we're about to get into system stability, it sounds technical and engineering-like and things like that. You got to have some mathematical ability. You really need none of that to understand this. I mean, it actually makes data far more intuitive and far easier to understand, even for those that aren't mathematically or statistically inclined, which I would count myself as being among those people.

0:19:41.0 Andrew Stotz: Yeah, it's interesting that point, when I think about what's fantastic about Dr. Deming and his teachings, was that he understood statistics to an extremely deep level. But then he brought it back to us in such a simple way of understanding common cause, special cause, and understanding that just chasing around data points is what most people are doing. And I just think that that's impressive because I studied statistics for a long time before I really thought about that type of stuff deeply. And I didn't understand that most of what we're doing is rewarding and punishing random variation.

0:20:23.3 John Dues: Yeah, that turns out to be the case. That turns out to be the case. So that third condition is understanding stability. So the thing to understand there is that systems can be stable or they can be unstable. And then from there we want to understand that by a stable system we mean it's predictable. We can within reason predict about how that system is going to perform over time. In an unstable system, we don't have that power of prediction. Unstable systems are unpredictable. The other thing to know is that if the system is producing predictable results, it's performing as consistently as it's capable. And so this third grade ELA or reading state testing system, it's a stable, predictable system. So there's no patterns in this data to suggest otherwise.

0:21:20.0 Andrew Stotz: Yeah. And I feel like, congratulations, you have a stable system.

0:21:25.9 John Dues: I mean, there is something to take from that. And it is that if you do have a stable system, it does tell you that what you do need to do is improve the system as a whole. There's no special thing to go look at. It's just that random variation that you were just talking about.

0:21:44.7 Andrew Stotz: And in this case, if we could just imagine on this chart that we take all of these lines and we shift them up without changing any data or anything except that the underlying data just is moved up so that the upper limit is 95 and the lower limit is 65 or whatever that would be, you would still have a stable system.

0:22:11.4 John Dues: Still have a stable system. Yep.

0:22:13.1 Andrew Stotz: But it would be operating at a different level.

0:22:16.6 John Dues: Yeah, you might not be as worried about it if it was performing at a much higher level. You might not have the same concern as a system that's producing a 60% average proficiency. So that's a good segway.

0:22:32.5 Andrew Stotz: In some ways, this chart and what you've explained is very depressing for us aspirational goal setters. Come on, let's get 20%! And what you see here is like Ah.

0:22:42.9 John Dues: Yeah, I mean, I think what I always say there is not saying it doesn't make it not true. I mean it is what it is. You might as well know it. Right? And that's a good point though, because just because this system is stable like it is, it doesn't mean it's acceptable. And we've talked about this before, but in this particular system, on average only 60% of the kids are proficient. So that means that only three in five students are reading proficiently. In this system, that means two in five are not reading proficiently. So we have this stable, predictable system, but it's producing less than desirable outcomes currently. So it might be a little depressing, but at least we know actually what's happening in the system. I think you have to know that before you can have any hope of improving it.

0:23:35.8 Andrew Stotz: Yeah.

0:23:36.2 John Dues: And then once you know those three things, the capability, the variation and the stability, then you can ask that question. You can ask that by what method question that Dr. Deming would ask.

0:23:48.7 Andrew Stotz: Yeah. And when I look at what this, as I was saying about the aspirational goal, all of a sudden it becomes harder. Like all of a sudden you come to this realization that to get to a new level of output of this system is going to really require new thinking, new action, rethinking. It's going to show that, there's so many things that hit me when I look at that.

0:24:15.2 John Dues: Yeah, yeah. I mean, so to drive that point home. So we've talked about that 20% stretch goal. It's beyond the capability of this third grade reading system as it's currently designed. The target is currently nothing more than a hope and a dream. But that all important question is by what method? So by what method could this third grade ELA system be improved?

0:24:40.2 Andrew Stotz: That great old song, coming in on a wing and a prayer.

0:24:45.5 John Dues: Yeah, that's about right. That's about right. But I think we should shift the focus to that question more than this question of setting a goal or people hitting this target that's been set for them in education. And we should think about what fundamental redesign of that third grade reading system would have to happen for any chance of improved outcomes.

0:25:16.1 Andrew Stotz: Yeah, I mean, this, by far, this wakes you up to that reality and helps you then start to rethink. How are we going to tackle this?

0:25:25.9 John Dues: Yeah, and you can do things like look for bright spots, right? There are places that despite having a high percentage of kids that are economically disadvantaged, still get very high scores. What are they doing that could be studied and perhaps brought to other places? Now you have to be a little careful there because education is one of those complex systems that context really, really matters. So you can't just pick up an idea from one place and put it in another place and expect it's going to work. There's more that goes into it than that. But at least going and finding those bright spots is something that's a starting place you can start to learn from.

0:26:11.5 John Dues: But that... At least when you have the three conditions, again, you have this logical starting point for that target setting. So again, you can set a stretch goal in my mind, but you should first understand those three things. And again, it should take on this improvement orientation where you've set this challenge out on the horizon, you've rallied your team around it, you've clearly communicated this is an improvement thing and not an accountability thing. And then you get to work as a team, slowly trying to figure things out and working your way closer and closer to that ambitious goal out on the horizon over maybe a six month or 18 month or three year timeline, something like that, depending on what the nature...

0:27:03.4 Andrew Stotz: I had an interesting experience with this because you helped me look at my student enrollment for Valuation Masterclass Bootcamp in a process chart. And I looked at it and I could see it was just like, this is a stable system. And then I made some adjustments to the system and we had a huge number outside of the control limits.

0:27:24.6 John Dues: Oh, yeah.

0:27:25.9 Andrew Stotz: And then the next time it went right back down where it was. And I realized like, it was like extra human effort for that period, like double down everything, and it wasn't a sustainable change to the system. And now I'm working on that sustainable change because I realized that even an exceptional effort in one period couldn't be sustained.

0:27:52.1 John Dues: Yeah, I think next time we'll actually talk about that. I'm working on an article about how to define improvement and how to know when it's happened and when it hasn't. So that'll be a good pickup point for next time. But I think that's the basic message I wanted to get across this time. And I think as we move towards wrap up, I think school leaders, I mean people in schools, they want to do better, right. So they set those stretch goals for that reason. But I think the three big ideas from this article can really help people as they're navigating that process with their team. So three big ideas that people can take with them from this episode. I think big idea one is that we've talked about setting stretch goals is ineffective and counterproductive in most cases.

0:28:54.7 John Dues: Big idea two was we want to gather that baseline data and then we want to plot it on some type of time series chart, even if it's just a run chart. And then big idea three was we wanted to understand those three conditions prior to setting any type of goal. And I think if people apply those big ideas to their goal setting work, now the team in place at the school is set up to answer that most important question, which is, by what method are we going to improve our system?

0:29:29.9 Andrew Stotz: Fantastic. Well, that's a great summary of it and I think you've walked through it in a simple and clear way. So I just want to thank you on behalf of everyone at the Deming Institute for this discussion. And for learners, remember to go to deming.org to continue your journey. And you can also find John's book, Win-Win: W. Edwards Deming, the System of Profound Knowledge and the Science of Improving Schools on Amazon.com. This is your host, Andrew Stotz, and I'll leave you with one of my favorite quotes from Dr. Deming: People are entitled to joy in work.