John Dues and Andrew Stotz are diving deeper into the improvement model that John is building with his team. In this episode, learn the three ways to think about an improvement frame for your big challenge.

TRANSCRIPT

0:00:02.6 Andrew Stotz: My name is Andrew Stotz and I'll be your host as we dive deeper into the teachings of Dr. W. Edwards Deming. Today I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. And the topic for today is Frame the Challenge. John, take it away.

0:00:23.6 John Dues: Hey Andrew, it's good to be back. Yeah, we're gonna talk about Framing the Challenge today. We kicked off a new series a couple of episodes ago. I introduced this improvement model that we can use to set ambitious goals backed with this sound methodology. Make this disclaimer again, we're sort of building the plane while we are flying it. So this improvement model is new at United Schools where I work. And so again, we're actually letting listeners sort of see it as it's being built and first put to use. And so I think just starting out with a quick recap of the model is a good place to start. So what is this improvement model that we've been looking at? I'll pull up my screen so we can share a visual of the model for those that are watching. Right. Can you see that all right?

0:01:21.2 AS: Yep.

0:01:21.3 JD: Great. Give me one second. All right. So we have this improvement model. Basically the core idea of the model is it gives us the scientific way of thinking. And remember, we talked about we're working to close this gap between current conditions in our organizations and future aspirations. In order to close this gap, we're walking through the four steps on the model. So first, we set the challenge of direction. That's really where we focused our time last episode. The second step is to grasp the current condition. The third step is to establish the next target condition. And then the fourth step is to experiment to overcome obstacles. And that's where we spend most of our time in this four-step process. And then the other thing we've talked about is we do it with this team. The people working in the system, that's one part of the system, one part of the team.

0:02:19.8 JD: And we've talked about this in our case. This can be students, it can be teachers, whoever the frontline people are in your organization. Then there's those with the authority to work on the system, to make changes to the design or the architecture of the system. That could be a teacher in a classroom, or we've said the principal of a school, or maybe the superintendent of a system of schools. And then one of the innovations that we've made to this improvement model is that that third group or that third person on the team is someone that has Profound Knowledge, someone that's using Deming's principles to guide the work. So that's the basic model that we looked at in the first episode and...

0:03:11.8 AS: And I would just highlight... By the way, can you put it on slideshow?

0:03:15.3 JD: Sure.

0:03:15.4 AS: And I've been reading Mike Rother's book, rereading his book on Toyota Kata and stuff. And so this has really got me back to it. But what you're doing is applying this and helping us understand it as you're putting it into action at your school, right?

0:03:35.0 JD: Yeah, we've had an improvement model. And I think... Yeah, so like it says down in the right hand corner, that this model, it's based on Mike Rother's work, the Toyota Kata work. I think one thing that was missing from our model previously was like, how do you set this challenge or direction? How do you do that in an ambitious but reasonable way? And I think Mike Rother's sort of model speaks to this. So that's why I like his four-step model. I also like the four steps because it's pretty simple. It's not 15 steps. It's not 20 steps. It's four steps. Now there are some steps that you have to learn and stuff like that. There are different pieces to each of the steps, but I like that it's four steps. It's sort of easy to remember. It's memorable. And I think the innovation that we've done so far is who is the team that's working through this? And I think to frame it as the people working in the system, the people working on the system, and then someone with Profound Knowledge, I think that's an innovation, from what I've read about Mike Rother's work.

0:04:39.6 AS: Yep.

0:04:40.4 JD: Yeah. And so to continue the recap from last time, I think in the last episode, so we introduced the model two episodes ago. And then last episode, what we said we were gonna do is start walking through each of the steps, episode by episode, and do a deeper dive into each of the steps. And we did that with step one last time. So we did Set the Challenge last time. And just as a refresher, this is that longer range goal that would differentiate us from other schools in our case, if we achieve it. But remember, we also said it seems nearly impossible at the outset. This is something that's off in the horizon. It's probably gonna take at least six months, probably more like two or three years. And then I gave this example at United where I am. So we're a school system. One of the challenges that we've set is to reduce our chronic absenteeism from 52%. So that's sort of the current condition. And we wanna bring that down to 5%. And there's this huge gap between those things. Obviously, we talked about an order of magnitude, and we don't quite know how to get there right now.

0:05:48.7 AS: Yeah, I think that's the point is we don't know how to get there right now.

0:05:57.6 JD: Yeah. Yeah. And I think... And so what I originally had planned to do is to go on to step two. And I think we're gonna do that next. But I wanted to pause because last time we briefly touched on this idea of Framing the Challenge as an improvement goal. And so with this episode, what I wanna do first before we go on to step two is talk about how to sort of think about that framing. And I sort of have studied a little bit of performance management from healthcare specifically. And there's really these three phases of performance management. When you're setting a goal, there's sort of performance measurement that has a research orientation. There's performance measurement that has an accountability orientation. And there's performance management that has an improvement orientation. And I think it's really important to understand the difference between those three types of measurement because I think conflation of those three things can derail and often derail improvement efforts. And in fact, as I was doing some research for this episode, I read this quote in one of the journal articles that I was sort of reviewing. It said, "the problem with measurement is that it can be a loaded gun, dangerous if misused, and at least threatening if pointed in the wrong direction."

[laughter]

0:07:19.6 JD: Right. So, when I read that I was like, this is important enough to take a pause and do a little bit deeper dive into that, you know, the differences between the three that we sort of got into on a surface level last time.

0:07:33.5 AS: One of the things that I like to say these days when I talk to people about measurement is measurement... If the subject being measured, let's say a table, I'm gonna measure the length of a table versus measuring the performance of an individual as an example. If the subject being measured knows, is aware that they're being measured, you're gonna have a problem.

0:08:02.0 JD: Yeah, I think that's very well put. And...

0:08:07.5 AS: The table doesn't care.

0:08:09.3 JD: The table doesn't care, but people always do. No doubt. No doubt. So I thought it'd be... I put together a table for those that are watching 'cause there is a lot of nuance to this and we'll kind of walk through this step by step, sort of the differences between goals or measurement for research, measurement for accountability, and then measurement for improvement. So I think just... I came up with these dimensions in some of the research I was doing. So, what's the purpose of each of those? What questions are you trying to answer? What are some example questions that are answered with that type of measurement? What's actually getting measured? How often is it getting measured? And then why does quality improvement or quality measurement matter in that particular area? So let's start with the purpose first. So when we're thinking about measurement for research, what we're really trying to do is contribute to some knowledge base, right?

0:09:01.8 JD: You know, I think the classic example is, what a university professor is often doing in their research. The second type of measurement is measurement for accountability. And really there, what we talked about last time is the purpose is to determine the application of rewards and sanctions or rewards and punishments. And that's really juxtaposed against measurement for improvement, which is... The purpose there is to learn our way to a system that produces a higher level of performance, right? And so let's look at measurement for research. We talked about the purpose being contributing to the knowledge base. If the questions that we're asking are about constructs or relationships between constructs or theories, then research is probably the direction we wanna go. An example question would be what's the relationship between two conceptual variables and what gets measured is... Could be numerous latent variables, but how often is this measurement happening? Typically once or twice during a study. And what we're trying to do is detect a relationship where they exist, right?

0:10:16.8 AS: So it could be... Like in a school, it could be a relationship between being late, the late rate and the absenteeism rate.

0:10:29.9 JD: Yeah. I mean, you could do research into why is it that student... Why are students chronically absent? You could do research into what's the best way scientifically to teach reading, right? And so you're gonna sort of come up with some answers there, at least answers that are sort of coming out of a lab, right? And a lot of times measurement for research or research goal or research study, that can be helpful to sort of initially point you in the right direction. You might do a literature review when you're trying to come up with solutions in your particular context.

0:11:10.3 AS: And it's important to remember that surveys properly done are a great form of research. So not only going back and seeing what's already been... What is the knowledge base on attendance, but also trying to do some research into what do students or teachers or parents think are root causes as an example.

0:11:33.3 JD: Sure.

0:11:34.8 AS: Okay. Great.

0:11:35.5 JD: Gives you a starting point, right? And so you certainly need measurement for research for sure. In that second bucket, we have measurement for accountability. And this is probably the thing that teachers and educators are most used to because there's accountability systems in all 50 states, right? And remember, we said the purpose is to determine who should be rewarded and who should be sanctioned. That's the purpose of an accountability measure. It's gonna answer questions about merit or status or accomplishment. It could be of someone like an individual teacher in a classroom, or perhaps about a school, for example. It's gonna answer questions like who's performing well and who isn't, who should be considered knowledgeable enough to do whatever, something X, right? But when we're talking about measurements, they're typically end-of-the-line outcomes, usually once per year after the fact. I've given state tests as an example multiple times. That's a very typical accountability measure end-of-the-line outcome. And why it's important to have quality measurement for accountability is that we can assign consequences based on measurement that lacks sufficient technical rigor so.

0:12:56.0 AS: Consequences as in rewards and sanctions?

0:13:00.4 JD: Could be... Yeah, consequences as in rewards and sanctions. And so there's technical guides that go with accountability systems. So how is the state, for example, calculating all of these different measures that show up on a school report card, test scores, value-added progress scores, chronic absenteeism rates? All those things have to be well-defined. Data has to be collected systematically. And it has to be done the same across the entire system so that rewards and sanctions are meted out equally amongst all the districts and schools and classrooms. But those two things are very different than measurement for improvement. And that's where I focus most of my time and where these talks really, really focus. And again, we said the purpose is to learn our way to a system that produces a higher level of performance.

0:13:48.4 JD: So we're talking about questions about specific changes as potential improvements to our systems. So some questions might be, are the changes I'm making leading to improvement? How are my changes affecting other parts of my system? And really, we're talking about outcomes and processes relevant to the object of change in terms of what gets measured. And that's happening... Those measurements for outcomes or processes are happening frequently as the practice or as the process occurs, right? Because we want feedback on a much more frequent basis than once or twice per study or at the end of the school year. That's one of the advantages here of measurement for improvement. And why does quality measurement matter in this particular area? Well, we wanna learn which changes are an improvement without wasting resources or will. Those are both very finite things in organizations, schools are... That's the same in schools.

0:14:55.2 AS: Will as in energy towards this objective, is that what you mean by will?

0:15:01.7 JD: Yeah. So two finite things, resources, which could be time or money. But will, I literally mean the will of the people, the will of the frontline people that you have to get on board with whatever this change is gonna be. And if you're moving between this thing and that, you sort of use up that will for good...

0:15:23.4 AS: It seems you have depleted the will of the people.

0:15:27.4 JD: And that happens all the time, especially where you're in a service business like education, the frontline people are being burned out all the time, teachers, in hospitals it's nurses and other folks in other industries. So that's the basic overview. And then I think one of the key things here is that there are some real measurement limitations when it comes to accountability measures and research measures or goals, when we're thinking about organizational improvement.

0:16:08.5 JD: I think the key limitation for accountability goals is that... The key limitation for improvement is that it does not illuminate why the outcomes occur or what should be done to change them when we're thinking about accountability system. For research, the key limitation for improvement is that it is impractical to administer it and not designed to inform changes in practice. So those are some real limitations. But what often happens, I think... And I should say again, like we said at the outset, that the three types of measurement are complementary. Like we need each of these three different types of performance measurement. But I think what happens is that problems arise when they're not used for their intended purpose. Remember, we said research, we wanna contribute to the knowledge base, that's the purpose. For accountability, application of award and sanctions. And for improvement, there we're actually learning our way to a better system, right? So, I thought it would be useful here. I may put you through like a little quiz here to apply the purposes of measurement to the right scenario. So, I have three situations here. They're unrelated to education. So, there's no pressure there. So, I'll read the three situations and then you're gonna tell me how would you... Which of the measurement purposes would you use?

0:17:47.7 AS: So, research, accountability, or improvement.

0:17:49.2 JD: Research, accountability, or improvement so.

0:17:53.6 AS: And I'm doing this on behalf of our listeners and readers so... And listeners and viewers.

0:17:55.7 JD: Everyone yeah.

0:17:56.7 AS: So, pay attention ladies and gentlemen, 'cause my answers may be wrong, but yours may be right. Okay.

0:18:01.7 JD: This is the check for understanding. This is a true education exercise here. And we're gonna be talking about avocados, right? So, there's no prior knowledge needed. So, I'll read through the first three situations, give you a chance to think, and I can repeat them if necessary, and you kind of think between those three. So, the first situation is rank the grocery stores in Columbus, Ohio, according to the quality of their avocados. So, would you use measurement for accountability, measurement for research, measurement for improvement? That's the first situation. The second situation is understand the relationship between weather, soil, acidity, and the eventual quality of an avocado grown in California. And the third situation is improve the quality of avocados on sale across all stores in Columbus. So, let's go back to that first situation. So, if you're gonna rank the grocery stores in Columbus according to the quality of their avocados, what type of measurement orientation makes the most sense?

0:19:15.5 AS: So I'm thinking accountability.

0:19:20.0 JD: Yeah, that's exactly right. Accountability, it's basically a grading system for avocados. You think how meat gets graded, it's grade A meat. That's really an accountability system.

0:19:33.6 AS: Okay, so listeners, viewers, did you get that one right? These are tough. John's a tough teacher. All right. Next one.

0:19:39.0 JD: You're one for one, and you have A grade schools, right? So you have A grade avocados, and that's an accountability measure. The second one was you're understanding the relationship between weather, soil, acidity, and the eventual quality of an avocado grown in California.

0:20:00.2 AS: Ladies and gentlemen, is this research, accountability, and improvement? Well, we've already eliminated accountability, so it's got to be either research or improvement. And if I get this one right, then I'm gonna get the third one right naturally. And I would say that sounds to me more like research.

0:20:15.8 JD: Yeah, that's exactly right. Research, right? So 'cause you're experimenting to see how the manipulation of variables, in this case weather, like the pH level of soil, acidity, impact the quality of an avocado. So you're basically a researcher trying to figure out what's the best combination of those things that gives you the best avocado. But this experimenting is gonna take probably years as you adjust those variables, right? And the last...

0:20:46.5 AS: Yeah, avocados don't grow so fast.

0:20:50.2 JD: They don't grow so fast, yeah. And then the third situation was you wanna improve the quality of avocados on sale across all stores in Columbus. There's only one left, so it's got to be...

0:21:03.5 AS: Well you used the word improve in it, so I think it's improvement orientation, huh?

0:21:06.8 JD: There you go. That's a giveaway. So aim is the quality of the avocado. So the basic theory of change is something like maybe improving the transport time from the field to the store. So there you can see that it's not like one is bad in terms of a measurement orientation and one is good. It's just... Is it being applied to the appropriate situation?

0:21:35.6 JD: There are certainly appropriate situations for accountability, appropriate situations for research and appropriate situations for improvement. So basically to sort of wrap this up, I mean, I thought it was really important because last time we talked, when you set the challenge and then it's gonna be something like a vision far out into the future, maybe two or three years, it's gonna be really important that that challenge is framed correctly because you're gonna be working on this thing for a long time. So with the model, we now have a way to bridge the gap between conditions and future aspirations. There's always gonna be a gap. We now have this model that gives us the scientific way of thinking and working to close the gap. And then we've said it's the responsibility of upper management to set this overall challenge as a key priority.

0:22:37.5 JD: And then we've said it's really important to understand the difference between these three types of performance measurement because conflation of the three can derail our improvement efforts. So the key takeaway here is you wanna frame this challenge or this far out direction that we're heading in as an organization as an improvement goal. And that's gonna orient the work. It's gonna orient the types of questions that you're asking. It's gonna orient the people, and the outcomes and processes that they're tracking. It's gonna orient your measurement system. You're gonna have to come up with frequent process and outcome measures that let you know how you're doing along the way. And the purpose of all of this is to learn our way to a better system. That's the purpose of measurement for improvement.

0:23:34.1 AS: That's great. A great summary. I wanna ask a question. Recently I've been teaching my corporate strategy course and I've been talking about the teachings of Richard Rumelt, who wrote the book called Good Strategy Bad Strategy, which is such a great book on strategy. But one of the things he complains about when he says his bad strategy is just setting an aspirational goal. Because as we learned from Dr. Deming, by what method? Like now, so there's... You've got to have a vision. You've got to have an aspiration of where you're going. But what he really focuses on... I think we're gonna talk about this in next sections. He really focuses on, have we really identified what the problem is?

0:24:21.7 AS: Like, what is the constraint? What is the thing that is holding us back from getting there? And what ends up happening is, when you clearly articulate the problem, what happens is it focuses... It becomes hard. Because to solve that problem, you need new resources. You need to get rid of old stuff. You've got to make substantial changes. And so it's much more comfortable for people to set strategies that are based upon wonderful visions. But never really deal with the problem. And then the workers in the companies, an employee in a company just looks up and goes, what did management just do for that long weekend? They did a weekend getaway to do their corporate strategy. And then they just came up with a fluffy vision of whatever with no help for us of how do we deal with the hardest problem is that we can't beat our competitor with the technology that we have. And we're never gonna get to that goal if we can't solve that problem.

0:25:28.7 JD: Yeah. And that's really the essence of the steps two, three, and four. I mean, the very next thing that we're gonna talk about is one, what's the current situation on the ground? Now we have this aspiration, but what's actually happening? For whatever data we have, whatever time period, 10 years, five years, whatever it is, we're gonna look at that data in a way that gives us a very firm understanding. And then remember, there's that crack in the model. That's the threshold of knowledge. So right up front, we're acknowledging.

0:25:58.0 AS: We don't know how to get over this. We don't know what's blocking us. We don't know...

0:26:03.3 JD: Yeah. We may have some ideas, but... Some initial ideas, but we do not know how to solve this chronic absenteeism problem. And that's where the experimentation comes in.

0:26:11.4 AS: So let me ask you one last question before we go. This is completely selfish, but also for the listeners and viewers out there, that is, I always ask my students this question in corporate strategy, should corporate strategy be kind of secretive in the sense that you're trying to build a competitive advantage and therefore some of the best battle attacks in war were kept secret. Stonewall Jackson was famous in his Shenandoah Valley campaign for not letting anybody know what he was gonna do the next morning. And then... Or should it be public? Like your number of 50 to five is pretty scary. And I'm just curious, what are your thoughts on that?

0:27:00.4 JD: Well, if you're in war, I think you should keep it secret. But pretty much anywhere else, I think public. Now that'd be my opinion. Now I'm in a school setting. I acknowledge that. But I've also seen...

0:27:11.4 AS: This is my quiz for you, by the way.

0:27:14.7 JD: What's that?

0:27:15.6 AS: This is my quiz for you.

0:27:15.7 JD: Your quiz for me. Yeah. Well, I have seen... And I've had the same thought. I think it was one of the founders of Toyota, or maybe someone that was an early CEO basically said, well, you give away your playbook basically. And he's like, well, just because someone has my playbook does not mean they can do it. And so education is a different orientation. There's not corporate secrets. And we're very open, we often share our practices. But I've often been on the...

0:27:45.2 AS: If you could get from 50 to five, you would want the world to know.

0:27:48.8 JD: I would want the world to know. I want everybody to do it. But I've been on the other side of this where I've gone to school and I've reached out to people get a manual or an artifact or something that they have that they do really well at their school system. And I look at it and I'm like, I don't know what to do with this. Right? So it's something altogether different to have the thing and then be able to do it. In our case, it's gonna benefit kids. We're gonna share it as far and wide as possible. So yeah, I think the setting matters. If I'm... The Union Army fighting Stonewall Jackson, then I'm gonna keep my secrets secret, my battle plan secret. But for most of the things I'm gonna share.

0:28:32.9 AS: Nathaniel Banks taking on Jackson. It was a rough series of battles for him. So I'll close out with my thoughts on this, which is that, yeah, I think ultimately once you've decided on what's your goal, where you wanna be, then I think you've got to make it public. And the reason why is because you need your employees to deliver that. It can't be... Everybody needs to be bought in. But even more importantly from a marketing and a relationship with customers, suppliers and others, they need to feel that vision. And they need to feel that mission. I think another great book is Start With Why. And that is why are we doing this? And I just read a great book on corporate strategy that is called Corporate Strategy Demystified.

0:29:25.5 AS: But it's just great because it was written in 2006. So he's talking about the battle between Apple and Compaq and Dell and all of these and IBM. And he doesn't know the outcome that Apple ends up being this multi-trillion dollar business. And basically his last sentence that I read in the last chapter is, it's over for Apple. They just can't compete in this space. And what he missed... This is what I'm teaching tonight, what he missed was the trusting connection that the customers had to Apple, to Apple's mission. Somehow Steve Jobs was able to create that mission and get it out to the world. And in the valley of death, when they were going through the worst time and it didn't seem like they were ever going to be able to do anything, it was customers that stuck with them because they believed in the mission of what they communicated. And it is that total, let's say intangible, that is very hard to measure and very hard to understand. But this is what I got when I was reading that book from 3:00 AM 'till 4:00 AM This morning, John.

0:30:44.4 JD: Whoever that author is was a small miss, a small miss.

0:30:48.8 AS: Yeah. And he's a brilliant guy. And so it's also a great point that just stick with your vision 'cause people's commitment to Apple and all that is so so strong. So being public about what you're doing and sharing it is critical because the last thing, as you said, even if somebody else has your playbook, I like to tell people that if somebody was working at General Motors and they had all the list of all the parts for Cadillac or whatever it is that they're building, and then they went to Toyota and said, build this, just because you have a list of parts and you have that operating system doesn't mean that they can build it because the product was actually designed for the operating system. And it's that entanglement in the actual process of production that makes that corporate strategy almost impossible to duplicate, even if you have the playbook.

0:31:41.8 JD: Yep. Yeah, that's it right there, I think. That's it.

0:31:46.5 AS: So fantastic. Well, on behalf of everyone at The Deming Institute, I wanna thank you again for this discussion. And for listeners, remember to go to deming.org to continue your journey. You can find John's book, Win-Win: W. Edwards Deming, The System of Profound Knowledge and the Science of Improving Schools on amazon.com. This is your host, Andrew Stotz. And I'll leave you with one of my favorite quotes from Dr. Deming. "People are entitled to joy in work" and in school.

Draft a top school MBA application in a week