Using Technology for Data and Feedback in Education

Quotas, arbitrary targets, work standards with numerical goals - these don't seem to apply to schools. But, as John Dues and host Andrew Stotz discuss, quotas show up a lot in classrooms, causing harm and preventing improvement.

TRANSCRIPT

0:00:02.4 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we continue our journey into the teachings of Dr. W. Edwards Deming. Today I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. This is episode 17, and we are continuing our discussion about the shift from management myths to principles for the transformation of school systems. John, take it away.

0:00:34.3 John Dues: It's good to be back, Andrew. Yeah we've been working our way through these 14 Principles for Systems Transformation. Last week or last episode we did eliminate slogans and exhortations. And so we're on to principle 11, which is Eliminate Arbitrary Numerical Targets. So I'll start with the overview. So principle 11, eliminate arbitrary numerical targets in the form of work standards that prescribe quotas for teachers and numerical roles for people in management, substitute leadership in order to achieve continual improvement of quality and productivity. And the first thing I wanted to start with was just this really powerful Deming quote on quotas. In Out of the Crisis, he said, "A quota is a fortress against improvement of quality and productivity totally incompatible with never-ending improvement." I just love that quote because it's just such a forceful pushback in the other direction. It's a fortress against improvement. There's really no gray area there in that quote.

0:01:43.0 AS: It's not a fort, it's not a barrier, it is a fortress.

0:01:48.2 JD: Fortress. It kind of brings together a mental image in your mind when you hear fortress, a fortress against quality, a fortress against improvement. So why did he say that? One of the things that's interesting is, especially thinking about work standards that prescribe some type of quota for teachers, it's like, well, when you think of a quota, you typically think of a worker and some type of production facility. And that, of course, is largely what Deming was talking about with his Point 11 'cause he was doing a lot of work in manufacturing and that type of setting. However it does, like all of this stuff, it translates into education. And, you know, so that's why I decided to keep Principle 11 'cause it does show up in different ways in the classroom setting. And I think examples are really good because when I initially read this quote and I was thinking, well, how do quotas show up in a classroom setting?

0:02:40.5 JD: And I thought of one that really stood out from when I was a principal here at one of our middle schools here in Columbus, at United Schools Network. We, we had this quota of sorts for homework. So in the middle school where I was, teachers had to assign homework nightly in their classes to students. And they had to grade two to three of those assignments a week and then return the graded assignments to students within 24 hours. And as I stopped and read this particular principle, and I thought about how it applied to my time as a principal, I really learned that that was the wrong approach. And as you start to think about that and reflect on it, you start to think about why Deming said quotas are a fortress against improvement. And there's this...

0:03:31.5 AS: And before you go... Before you go on, I just wanna highlight how normal that sounds.

0:03:38.4 JD: Very normal. Yeah. Very normal.

0:03:39.8 AS: And anybody here, like the first thing I'm gonna do or the first thing I do when I take over as principal is I'm gonna require that there's a minimum amount of this, and it has to be da, da, da, da, da. And it just seems like it is the responsible thing to do as a manager.

0:03:56.4 JD: Yeah. And there was a noble premise behind the quota, and that was that students needed frequent feedback on their work in order to learn. That was the premise, right? So it was this work standard, it was well-intentioned, but like a lot of these things that Deming talks about and quotas are no different, is the actual effect was that teachers spent less time giving feedback and more time grading this high volume of work. So this is what happens when you have a quota, is the focus became meeting the quota, grading the two to three assignments per week, rather than giving that quality feedback to students. So in this case, this sort of numerical target for graded assignments, then superseded the quality of the feedback. And this is, this is what happens, I think, generally speaking, with quotas and practice.

0:04:56.3 JD: And so when you step back and you think about that particular quota, you say, well, what was the teacher's job? Was it grading two to three homework assignments per week or was it giving students quality feedback? And it really couldn't be both. It couldn't be both those things. So then I started thinking about, well, what would've been a better approach? You know, had I had the Deming lens when I was a principal, I think the thing that I would've done is start with, well, let's come up with a sort of a well articulated aim for why we give homework. And included in that process, or included was, developing a process so that students received timely and high quality feedback 'cause that's really what this was about. So I think that sort of brings to mind substituting leadership, that second part of the principle.

0:05:58.4 JD: And in this case, I think, you know what I should have done is replace those work standards or that quota with some type of a better understanding of the job of the teacher. So, you know, I think in doing that, then I also... Leads to higher quality work-life for teachers. I would say, in thinking about this homework example, time spent grading homework was probably the number one complaint that I got from teachers. And this better approach to the two, three, assignments per week quota would've been to work with teachers to design a better system. Like how could we design a system that would give them the time to deliver high quality feedback to students on a timely basis? That was really the aim. And that's really where I should have concentrated my time as a principal. But again, I didn't have that Deming lens 10 years ago when I was serving as a, as a principal in our network. And reading the Deming stuff, it was very quickly like, oh, aha. Like, here's how I should have been thinking about this. Rather than being so hyper-focused on: you gotta grade those two to three assignments every single week.

0:07:29.9 AS: I like the word substitute leadership, you know, and Dr. Deming said that a lot. And the best way that I've kind of tried to explain it, and it just happened recently, where a client of mine was talking about having what they would consider to be underperforming staff. And they were older. They'd been with the company for a long time, and like the mindset is not there. And so their idea was to use KPIs as a way of basically catching these people out and then eventually firing them from the company. I'm making it kinda crude, but that's kind of the way it came across.

0:08:09.7 AS: Yeah.

0:08:10.1 JD: And I was like, wait a minute, let's just get down to the meat of this. The fact is, is that you hired these people [laughter] and you led these people for 20 years, who's responsible for this? And then I said, look, don't substitute leadership... Don't substitute KPI for leadership 'cause people say, so if I don't have KPIs or I don't have this, how am I gonna manage the people who aren't performing? I'm like, you know the people who aren't performing, they're probably in the wrong job. They may be in the wrong company, they may be the wrong thing, I don't know, but you need to talk to them and work it out and figure out a solution, that's leadership. But hiding behind some sort of quota or target and thinking that that's gonna solve the problem, no, that's why we need leadership.

0:09:06.4 AS: Yeah. And knowing the staff that I was working with at the time, the group of teachers, I am sure I am 100% sure that we as a group could have come up with a way, a better way to do our feedback system than the way it was set up. I have, I have no doubt. If we said, look, this actually isn't really working that well for teachers the time it's taking just to grade the homework. The kids, the students are, do they really need homework every single night? And when they're getting these papers back in the morning, do they have any time to actually look at whatever feedback is provided? Sometimes it was pretty minimal. Sometimes there was, depending on the teacher and the assignment, sometimes there was some feedback there. But are we giving kids time to look at that and actually learn from that feedback in any way? And so, again, you know, well-intentioned as it was, the volume superseded the, you know, the quality of the feedback. So I can think of all types of ways that I would sort of redo that system in retrospect with a clear aim is where I would start, what's the aim of this? Whether it's homework or classwork or whatever it is.

0:10:23.9 AS: And with technology now too, it's just such, it's gotten a lot easier. Such as give the students a five question online quiz that tests the topic that you taught that day. Then accumulate the data and understand what was the hardest one or two questions. Then in the first 10 minutes of class or five minutes of class, say, okay, last night's assignment, the hardest question was number three. And now I'm gonna randomly select one person to tell me how did you answer number three? And then let's have a discussion on that. And then that way you're getting feedback. It's the same thing I did with Feedback Friday just 'cause you were talking about feedback. Where everybody wanted, they requested in my Valuation masterclass bootcamp, they requested more feedback and I designed Feedback Friday where I gave them the exact assignment, then a certain number of them will present their work, the ones who volunteer in this case, and then they present their work on Friday. And then I give feedback that everybody witnesses and can learn from.

0:11:44.8 JD: Yeah. I mean, I think there's so many different ways, like what you're describing to set up the practice, to set up the feedback system, to have students pair up or someone present their work or you know, there's all types of better ways that would would've saved a lot of people, a lot of time, a lot of headache. There were many, many ways we could have redesigned that system.

0:12:06.4 AS: And why use the word arbitrary? You've said eliminate arbitrary numerical targets. We've talked about the numerical aspect of it, but why do you say arbitrary?

0:12:19.5 JD: Yeah. I wrote an article that actually called them arbitrary and capricious goals. So not just arbitrary, but also capricious, but yeah.

0:12:26.8 AS: What does capricious mean? How would you define it?

0:12:28.4 JD: Well, it's sort of the same as arbitrary. It's sort of like without any sort of grounding in logic or reality. [laughter] Flippant, sort of, I think I have that right. You can fact check me on that definition. But that's actually a perfect segue into that 'cause there's that second part of the principle that talks about also eliminating numerical goals for people and management. So not just teachers, but also school or network leaders. And we've talked a lot about, and across the series about various types of targets that exist in education. But I think it's still worth discussing a few points. Like what does arbitrary mean in this setting?

0:13:07.2 AS: And capricious means "given to sudden and unaccountable changes of mood or behavior." [laughter]

0:13:12.5 JD: Yeah, and I think in that article I've used capricious because in education, you know that the targets are changing so often, especially associated with test scores or other sort of parts of state accountability systems. One year it's some type of label, another year it's letter grades that schools are rated on. Now in Ohio we're given star ratings. So they've gone away from A to F and now it's schools get rated based on a five star system. And so that's what I mean by capricious. It's just...

0:13:42.4 AS: So five is A?

0:13:44.5 JD: Five is good. Yeah, five is good. Five is an A, I suppose. If there's a difference between a star, a five star and an A, I don't know, [laughter], but that's the new thing here in the Buckeye state. Yeah, but so back to the management. I think, well one thing I think when you're talking about teams or departments or a school or the whole organization, I think they should have an aim. So I think that's really important. But by aim, I'm talking about some type of clear purpose statement, but that's not so specific in detail that it stifles initiatives, that's one part. And I think a clear aim statement sort of in that spirit is something very different from a numerical goal. So that's one part of it. I think another part, and I'm getting to the arbitrary part, is that so many times that internal goals either set for management or set by management of an organization are basically a burlesque,you know, if they don't include a method.

0:14:52.8 JD: And that's what I've seen so often where, we've sat down to create a scorecard or whatever, or we see these things imposed on us from some type of organization that holds us accountable like the state department is, we have these goals, like increased student attendance rates by 10% or increase math scores by 5%. And the thing is, is part of the arbitrary is that sort of natural variation in those attendance rates or in those test scores are viewed as a success if they're going in the direction of good, but fluctuations in the other direction sort of send everybody scurrying around looking for explanations. And we end up sort of writing fiction to explain away or to explain causality when there's not actually causality there. And I've been just as guilty. We're gonna improve test scores by 10% next year without some clear plan.

0:15:53.7 JD: But then what Deming would say is, if we can do it next year with no plan, why didn't we do it this year? [laughter] And why did we stop at 10%? Why not just make it 20%? That's again part of the arbitrary. Now there is a caveat and Deming I think makes this caveat Out of the Crisis that's important. And that's when... When we are setting numerical targets or talking about numerical targets that he categorized as, he called them, facts of life. Meaning, they're just sort of plain statements of fact with respect to survival. So a good example would be, for a school would be, unless student enrollment improves by 10% next year, the school will have to shut down. But that's not arbitrary, that's a fact of life.

0:16:43.3 JD: If we don't get the revenue associated with increase in enrollment, then we're gonna have to shut down. That's not an arbitrary target. That's something sort of something, you know, altogether different that we should pay attention to. But I think getting to this idea of arbitrary, I would probably characterize it this way, and this is probably the main point of principle 11 for leaders is that they have to understand system capability. And that's often what's sort of missing from the understanding when these arbitrary targets are set. So if the system is stable, you'll get what the system will deliver. If the system is unstable, then there's no way to predict capability. So I think it's fine for organizations and individuals to have goals, but the problem is so often just like what you zeroed in on is they're arbitrary.

0:17:50.8 JD: And that's been my experience during the vast majority of my career is, why is this target set at 80%? And then no one can really tell you why that was picked as the target. For whatever the thing is, I just picked 80%, sometimes it's 75%, sometimes it's 80%, sometimes it's 90%. But I think that type of goal setting is really inevitable when as a sector, we have really no understanding of the theory of systems, we have no understanding of the theory of variation. And so without that understanding, what we tend to do is blame individuals working within school systems instead of working to improve the system itself. And I think that's really the key for me and why I'm so zeroed in on the Deming philosophy 'cause it does offer this other way. And so thinking back to system capability and before we as leaders set goals, there's really four kind of things that I've tried to focus on. And one of them is: what is the capability of the system or process under study?

0:19:13.3 JD: And when I say what is the capability, the easiest way to think about this is, if I display my data, let's say it's third grade reading state test scores. By capability, what I mean is, if I've look at the last dozen years of those third grade test scores, there's gonna be an average. And that average of those twelve years is basically the capability of my system. So I'm not just looking at last year or even the last two years, I'm looking at a dozen or more data points preferably, to see how capable that particular system is. So that's sort of the first thing I'm gonna look at is, what's the capability of that particular system or process over time? The second thing I'm going to do is ask, what's the variation within that system? So we've talked about the process behavior chart a lot. I do think I'm sort of in the Donald Wheeler camp and thinking that that is the most important tool. And so it's very easy using a process behavior chart because I have the limits, the upper and lower limit.

0:20:29.8 JD: And however wide or narrow those limits are, that's gonna tell me the variation I can expect in that particular system over time. The tighter those limits are, the less variation, the higher the quality, or at least the higher the predictability of that system or process. The third thing I am gonna wanna know is, is that particular system or process stable over time? Do I see any signals in the patterns in the data that would say that this is a stable system and therefore it's predictable or is it unstable and therefore it's unpredictable? So that's the third thing I'm gonna ask. And then once I've answered those questions, the last thing I'm gonna say is, do I have a logical answer in thinking about whatever goal we're setting to the question by what method? And so I think if you don't have that sort of picture in your head, the goals that we set are sort of arbitrary and capricious. If we are setting a goal, we can answer those four questions, then the goal is probably reasonable, logical, and grounded in some type of understanding of our systems. That's what I think Deming meant when he talked about arbitrary targets or arbitrary goals.

0:21:58.1 AS: So let me review that for the listeners out there. Number one, what's the capability of the system? Number two, what's the variation of the system and understanding a process behavior chart? Is the system, number three, is the system stable over time? Is it predictable? And number four, do I have a logical answer to the question by what method? And what you get from that is that, clearly if you can answer those questions, you understand your system pretty well. And therefore it's less likely you're gonna come up with an arbitrary goal. You're gonna go to the, say, here's what I think we can do with a deeper understanding of the system. But if you have a bureaucrat from the state education department, as an example, say, I want 5% more. Why not 5% less? [laughter]

0:22:50.4 JD: Right. Based on what?

0:22:51.4 AS: But where? Where does that come from?

0:22:53.5 JD: Yeah. Yeah. I think another thing I mentioned Donald Wheeler, he said, goal setting is often an act of desperation. And I think I talked about earlier in this series of episodes, I've talked about third grade reading test scores in Ohio. The goal is 80%. The system right now is capable of about a 60% in terms of looking at the system of third grade reading test scores in the state of Ohio. So 80% is a hope and a dream. It's somewhere out in la-la land. You know, and and what's happening is schools are being held accountable for that number in a system that is not capable of meeting that target, far from it, as a state. So I'd wanna know, like who set that and on what basis was that goal set for third grade, for third grade reading in the state? And that sort of thing is happening over and over and over and over again. What's the latest thing that we're gonna focus on? I think chronic absenteeism is one of those things right now that everybody's talking about. Kids aren't coming to school like they did pre-pandemic without any understanding of the theory of systems and the theory of variation. And so people are just running around talking about it without any understanding of what that data looks like over time.

0:24:27.2 AS: Yeah. And I'm looking at Donald Wheeler's goal is often an act of desperation, part one. And some of his discussion on that is great, great stuff. I was thinking about, in my own case with my Valuation masterclass bootcamp, when I first started the bootcamp, now we're on bootcamp number 12, but the first ones, I just told the students, okay, pick any company and then you can write a report on that company. And the outcome of that was disaster. Like it was just so... And I realized I didn't have that much teaching involved in how to get them to where I wanted to get them. So I had to... First I had to start to improve my teaching knowing that I'm trying to narrow the outcome to be, you know, somewhat consistent. And then I realized I can't just let them do any one company by choice, I have to kinda give them a list 'cause there's a certain companies that just don't have much information.

0:25:23.3 AS: If they choose it, it's a bad company for them to work on. So then I would give them a list of a hundred companies and say, pick one out of this, and each person had a different company. So that started to improve it that I had more information. And so I'm iterating through this and then I realized, some people just state that it's harder for them to do this assignment. It takes six weeks to do it and it's just overwhelming for some people. And I thought, what if each team did the same company? And I assigned it. And so what I did is I set up teams and now I encourage the teams to work together. They each wrote their individual report on that company, but now they start sharing information. And now I'm narrowing down, and I'm getting my system more and more narrow and the outcome is getting more and more narrow.

0:26:07.1 AS: And then I have a deadline that by the end of the fifth week, you've got to submit your draft. If it's not up to the standard, I can't put you in a time slot. We're gonna have to figure out something else to do.

0:26:21.5 JD: Yeah.

0:26:22.0 AS: And so that prevented someone who's postponing until the last minute, we're giving them a deadline that's a week before to give them some time to wake up and fix, make sure they got the stuff fixed as much as they could.

0:26:35.6 JD: Yeah.

0:26:37.4 AS: And then recently, so we've been iterating through this and improving the system. And now the outcome is better and better and better of what they're doing and the way that they're presenting. The way that I'm able to... I can't teach about a hundred different companies, I can teach about a few and help them in that process. And then in this particular bootcamp, my idea for improvement was what if we... Instead of starting the first week by assigning them that company and the team gets all excited and they start working on that company, let's say the company's Tesla, as an example. Why don't I pick an industry? And in the first week, everybody in the bootcamp works on an industry report, which is just a one-page report. What are the key features? And now everybody's going out and getting industry analysis, third-party research, and that's helping them.

0:27:26.1 AS: And I had four groups that are later going to be assigned the actual company. So we looked at the automotive industry and then there'll be assigned companies like Ford or Tesla or Toyota or whatever. And so by the time we get to the second week, they've now got a really good picture of the industry. And all of a sudden it adds a lot. My hypothesis is it's gonna add a lot of context to their assumptions in the final report. The reason why I'm explaining that is the idea of a process, a system. And in this case, I have to say, I'm not like measuring it very, very specifically, I'm judging the outcome based upon my experience in the prior outcomes. But if somebody came along and they said, hey, why don't you improve this? Why don't you do this with the system and set an arbitrary goal? They would have no understanding of what we've been through, what we've learned, how we've iterated through it. And without that understanding, almost anything would be arbitrary. They're interesting ideas and I listen to what people say, but almost everything would be arbitrary in a system that you're not studying or that you are studying in detail.

0:28:33.9 JD: Yeah. I also think about our previous conversation about, I don't know how you evaluate the final project, but I think of the Deming admonition to abolish grades. And you can very clearly picture if you... How many cohorts have there been of the, that class?

0:28:53.8 AS: We've had 12.

0:28:55.1 JD: 12. So let's say you lined up the sort of reports, a representative sample from cohort one to cohort 12, and you looked at the quality of the reports from cohort one compared to the quality in cohort 12. It sounds like because of all these iterations you've done, the quality is much higher in cohort 12 than cohort one. But then the question would be, who do you assign that evaluation of that grade to? See what I'm saying? Like now you can start to see why Deming said that. Because the lack of quality or the lower quality in earlier cohorts is as much attributable to you and what you were doing as the instructor as it was to the students.

0:29:42.8 AS: It was almost 100%.

0:29:44.5 JD: Yeah, so you can see... [laughter]

0:29:45.0 AS: Because I was setting the whole system. And I think that's where you get the idea of substitute leadership.

0:29:54.3 JD: Yeah.

0:29:55.7 AS: Like it's you, for the listeners, for the viewers out there, it's you, it's your responsibility. The outcome is your responsibility. The outcome is an outcome that's happening because of the system that you're running and participating in and operating. And I could go back and look at my bootcamp number one students and go, they were terrible. [laughter] But the fact is, there's no difference between the raw material that came into the bootcamp in the first group versus the one that came in in the 12th group, but they're just so... The output's so much better, so you can't argue that it's the students, it's an improvement in the system. And let me just add one thing about grade. I really don't know how to grade them truthfully. So what I just say, first, you've got to hit that deadline of having a good quality draft. I said, it doesn't have to be perfect, but it has to show that you've put time in. Otherwise, I can't spend time in your presentation with you during the final presentations, which means you're not gonna graduate. You can come back and try it again, and we can do it in another way.

0:31:03.4 AS: We'll talk about that later. But then the second thing that I do is say, if you can submit on time and you can present on time and according to the guidelines that we give, which they can do, then you pass. So I guess it's kind of pass/fail. Now, what I do is I pick out what I think was the best one of that particular cohort. And I have never announced that, and I've thought about it in my team. My other team members have said, wait a minute, what about... All you talk about a Deming, and here you are highlighting this one person and all that. And I was like, yeah, that's a good point. So we haven't really done it, but what we did do is take that one and we use that in the next cohort to say, this is the high bar. That this is one of the best ones that was done in the last group and my goal is to have you exceed that.

0:32:03.6 JD: Yeah. Yeah. To me it sounds like sort of what you've been doing to improve that particular, that class, that system, follows the Deming philosophy to a T, really. Yeah. Yeah.

0:32:21.4 AS: And I think in the end, I think the key thing and maybe we'll wrap up on this is just the... I like the idea of the arbitrariness because what it tells you is that really to set, as you said, Dr. Deming didn't particularly, he's not against goals and he's not against plans and all that, but it is that arbitrary nature of somebody just coming into a system that they didn't really know much about and setting some arbitrary goal. And really that just disrupts the system. And so for the listeners and viewers out there, if you are setting some arbitrary goal without having a clear understanding of the system, then what's holding the system back could be you. And that to me is a big takeaway from this. Any last thing you would add?

0:33:08.3 JD: Yeah, and I was looking at that article that I wrote. I was in a series called Goal Setting is Often an Act of Desperation. And the definition I used for capricious came from a law dictionary 'cause that's where capricious most often shows up is in the legal world. And it's "a willful and unreasonable action without consideration or in disregard of facts or law." And so that's what I was feeling is that often what's happening is educators are given these targets that have no basis in reality and that can only cause consternation and we're seeing churn. And so, and the people that work in education, teachers leaving and those types of things, and I'm not saying it's all for this reason, but it certainly doesn't help when you're constantly being given goals that are not set in reality. So I think if we took those steps to do those four things when we're setting goals, what's the capability, what's the variation? Is there stability and do we have a method? I think we'd be far ahead of where we are now.

0:34:09.8 AS: Boom. John, on behalf of everyone at the Deming Institute, I wanna thank you again for this discussion. And for listeners, remember to go to deming.org to continue your journey. You can find John's book, Win-Win, W. Edwards Deming, the System of Profound Knowledge and the Science of Improving Schools on amazon.com. This is your host Andrew Stotz and I'll leave you with one of my favorite quotes from Dr. Deming. People are entitled to joy in work.