The Problem with Grading on a Curve and its Consequences

Most of the time, variation between students or workers is the result of common cause situations, but sometimes you find folks who consistently aren't performing at the same level. Does more punishment work? What should you do instead? In the episode, David Langford and host Andrew Stotz discuss how managers (or teachers) should approach these "special cause" situations.

TRANSCRIPT

0:00:02.4 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we continue our journey into the teachings of Dr. W. Edwards Deming. Today I'm continuing my discussion with David P. Langford, who has devoted his life to applying Dr. Deming's philosophy to education, and he offers us his practical advice for implementation. Today we continue our discussion of Dr. Deming's 14 items that he discusses in his New Economics book about the role of a manager of people after the transformation. This is on page 86 of the third edition, or page 125 of the second edition. And this is point number nine. Let me read it to you before we get started. So again, for a, the role of a manager of people, this is the new role of a manager of people after transformation. Point number nine, he will try to discover who, if anyone, is outside the system in need of special help. This can be accomplished with simple calculations. If there be individual figures on production or on failures. Special help may be only simple rearrangement of work. It might be more complicated. He in need of special help is not in the bottom 5% of the distribution of others. He is clean outside that distribution. And Dr. Deming presents a normal distribution and some other things, in this chart that he presents in this one. And we're gonna call this episode: Who Needs Special Help? David, take it away.

0:01:40.5 David Langford: Okay. Yeah, this is always a topic of discussion because, there's all kinds of management theories out there, right? About, how we manage, I can't remember who, was a proponent of just getting rid of the bottom 10% of your...

0:01:57.9 AS: Jack Welch.

0:01:58.0 DL: Organization every year. Jack Welch, yeah. Notoriously wrong, with that. And, or well, "if you can't cut it, out you go." And that all sounds good until it becomes so expensive to constantly be hiring new people and replacing people. And the fear level goes up so high that you can't get anything done because nobody wants to take any risks because you really can't take a risk because you might be gone. So Deming is saying a lot really in this point, he talks about the distribution of people. Well, so first thing is you have to figure out what is that distribution, right? So how are you calculating that? Or how are you figuring out what that performance level is? Well, as a teacher in a classroom, obviously, you have tests that you're giving, you have projects that are happening, etcetera. It is actually pretty easy to see that distribution of performance in a classroom. You give a simple test on something and then you look at the test results and you start to see, okay, everybody scored on this test from 70%-100% on this test, right? So you can say, okay, that's an average of about 85 or so for the whole class.

0:03:31.5 DL: When you look at it on a histogram scale like that, what Deming is really talking about, he's not talking about just the people that were scoring at the lower end of that distribution. People that were getting 70, 75, 80, etcetera. They were all at the lower end of the distribution of that system. But what it's showing is that's the capability of the system. You did something, you did a process with people, you tested the process, the process produced that curve, and on average, it gives you an average of 85. Now deciding whether or not that's good, is good, is good good enough, that's a whole different really discussion than what really Deming is talking about here. So he's not talking about people just on the lower end of a distribution of performance. He's really talking about somebody that's completely outside of that distribution. So in a classroom, if I did something like that and we did a project or a test or whatever, and everybody is scoring from 70%-100% except for maybe two people that got 10 or 5, right?

0:04:46.6 DL: Obviously these are two people completely outside of the system. And what he is really talking about is probably no amount of adjusting the system is going to help those two people. They are so far outside of the distribution that they really do need special help. So in a classroom that can mean, this could be children with special needs, they could be hearing defects, they could be the eyesight that, I don't know how many times I thought somebody was an understanding problem. And then we find out, oh, they couldn't see, either they couldn't see to read or they couldn't see the...

0:05:31.8 AS: Something very simple.

0:05:32.3 DL: The whiteboard in front of them. Yeah. And they got tested and got glasses and everything and wow, it just made a huge difference. But obviously when you have people in a special category, it's gonna take much more time and effort individually to deal with them. Right? And that's why it's called special, special needs, right? Because you are gonna take the time and effort individually to deal with those individuals. If you don't have anybody completely outside of the distribution of performance, then you're gonna go back and look at the system itself. So in my example, everybody is scoring from 70%-100% on some test that you give them. And the average is 85. Then you have to decide is good good enough, is that a good enough distribution on this? And as a teacher a lot of that has to do with understanding where does this fit in the entire curriculum.

0:06:40.2 DL: So is this a critical skill, that if these students don't have this skill and they don't have it just down pat, and are acing it, it's gonna cause huge problems later on. So it might be worth the time to go back and sort of rework that for the entire class and see if we can get a higher average. On other things, you might look at that and say, "Oh, okay. Only I know really the whole curriculum for the year, and I know that we're gonna be revisiting the same concept probably four more times throughout the year. So this average at this time of the year is probably good enough." I often joke with teachers and say, "If you're happy with your average, and you know it, clap your hands," so.

[laughter]

0:07:31.0 DL: But if you're not happy with your average and you know it, then you have to think about, okay, well what am I gonna do about it? Do I have the time to go back and rework this? And Deming in his example in figure 12, that he's showing there, is actually talking about moving in the entire system forward. So shrinking the variation so that it's not nearly as wide as it used to be, and more people are getting a higher average within that. So how do you get that higher average? Well, prevention is the key to quality. So every time you're doing a lesson that you've done before and you're taking that feedback that you've gotten before, folding that in. And this time when I did it, ah wow, we got an average of 89, or we got an average of 93, or...Excuse me. It's really difficult when you're improving a system and you're moving that average up, each time you go through something, when you start to get up and really high levels performance, going from 93 to 94 is a really big effort.

0:08:57.4 DL: There's gotta be something really happening there to get that next level result. And do you really have the time right now to get that? Or is it a problem of tomorrow that we have to figure out, okay, what are we gonna do in this system, in the future, to get a higher average? But I didn't believe, really, this when I started working with Deming, but then I went back and looked at all the grades and scores that I had given people, and I was so predictable. Every year I had the exact number, the same number of people getting A's or doing top-level work. I had the same level of percentage of kids that were failing I had... But of course it was always their fault, not my fault, so. And so that was really eyeopening to me that all I had been doing is just basically, for five or six years I had been doing the same thing over and over, and over, and expecting a different result. And it just doesn't work like that.

0:10:05.2 AS: Yeah, this one is interesting because first of all, he's presenting us with a distribution. We can see a normal distribution, and he's presenting also a more narrow distribution, saying that the goal is to try to, maybe in this particular case that he's showing to, he says, "You wanna work to improve the system by narrowing that distribution so that..." And shifting it, as we can see, as we've talked about. But I think also in this one, if you don't understand the system, you can get caught up in chasing performance in individuals that actually are just a normal outcome. And you miss the time that you need to spend to fix that special cause that needs to be fixed. So that was one of the things I took away from it. What do you think about that?

0:11:02.6 DL: Yeah, that's why he hated practices like grading on a curve. Which is notorious, it is still is notorious in many universities, grading on a curve. And... Because that shows no understanding of performance and distribution and average performance and it takes no accountability. And for you as the teacher, it just all blame on the student. "Well, if you tried hard, you could do that." Well, no, that's not true. There's only gonna be so many A's, so many B's, etcetera. So you're not gonna ever get there, so. But really this is about... Go ahead.

0:11:43.8 AS: To understand that a little bit more, so is the problem about grading on a curve that you're constantly... You're not necessarily improving? You're just like, "Well, this group had a curve that was here on the continuum and this group had a little bit better, they were better." And what is it? 'Cause I'd say grading on a curve is something that people on initial blush would think, "Isn't that what Deming is talking about?" I mean, we see normal distributions, we see curves. Explain that in more detail.

0:12:15.1 DL: You're creating an artificial scarcity of top marks. So only... No matter how well we do as a class, there's only gonna be so many top marks or people that are gonna get the top grade, right? And so you're gonna create all kinds of competition and you're gonna create all kinds of weird behaviors that go on. You're actually encouraging people to cheat. And I can't remember if I told you this story or not, but one of my children, she was in a advanced chemistry class or something, I think it was. And she comes home the first day of school and she said, "Dad, I think this teacher would be really interested in talking with you about what you do and improvement, and everything else." And I said, "Why?" And she said, "'Cause he said, well, everybody in here can achieve, everybody in here can get an A, can do well."

0:13:14.1 DL: She comes home the second day of school and she said, "I think I'm gonna drop this chemistry class." And I said, "What happened in two days?" She said, "He came back today and he spent the whole hour of the class explaining how he grades on a curve. So there's no way in the world that everybody in here is gonna get an A," right? You're creating an artificial scarcity of top marks and it's just not gonna happen. And I said, "Okay, well, just let me know what you decide to do with that." Well, she comes home the next day and she said, "I think I'm gonna stay in the class. I'm pretty sure I'm gonna be one of the people on top of the curve." And this was an honors chemistry class and in that class, half of the kids in that class had had straight A, 4.0 averages to that point. So, there was a bunch of kids that quit, 'cause they could not risk getting even a B in a class like that.

0:14:18.0 DL: But my daughter stayed in the class, at the end of the first semester, she comes home laughing one day and she said, "Dad, you'll never guess what happened." I said, what? And she said, well, this is a very, very smart group of kids. And not only did kids keep track of their own scores, they actually kept track of other kids' scores in the class as well. And I think there were one or two kids that found out that there were a bunch of kids that were just right on the line between a B and a C or something. But if those kids failed, it wouldn't make any difference to them. They're still gonna get the same grade at the end of the semester. Even if they didn't even take the final, it's not gonna affect them one way or the other, they're still gonna get that B or a C grade that was in that. But if they did fail it would mean it would change the curve and these other top kids could move up into the top echelon.

0:15:13.3 DL: And so they paid these kids $20 to fail the final. Well, somehow the teacher found out about it and then the principal found out about it. And there was a Spanish inquisition that was taking place and then they were talking about expelling kids and all kinds of stuff, I couldn't stand it, I had to go and talk to the principal and I said, how do you like it? He said, what do you mean? I said, "They're better at managing your system than you. They figured out how to play your game better than you. And you gotta be rewarding these kids not... And recognizing amazing statistical analysis and capability, not punishing them through that process, so." I think it was the same principal that said, "I know I'm having a bad day when your car is in the parking lot," so.

[laughter]

0:16:06.6 AS: Exactly. You should have said, you should have been... You didn't even realize you were teaching 'em a double major AP chemistry and AP statistics.

0:16:17.6 DL: Yeah, absolutely. So.

0:16:19.3 AS: Well, let's wrap this up by... I think the key thing of what Dr. Deming is telling us in this is about understanding your system and then identifying if someone is outside of the system, and that person or result outside of the system is... You know, warrants some special attention or special help and that that, you can't really know that without understanding the system and also not being too distracted by the variation that's natural from that system. And therefore, ultimately, once you understand that, then you really can clearly identify that some outcome or some individual is a special cause and then you can focus in on that and fix it. And so that's how I would summarize it. Is there anything else you'd add to that?

0:17:12.9 DL: Yeah. I was just was recalling that you are... Deming explaining several times that if somebody is outside of the systems that, far outside of the system, further rating and ranking are not gonna help them at all. Giving them more failures, more Fs, docking their pay. Whatever you're thinking of doing to somebody that's completely outside the system it's really not gonna help them at all in that process. And that... That's not help. Rating and ranking and bribing people to do better is not actually helping them. You actually have to study cause or the reason why that person is special cause and then do something about it. And in a classroom, and it could very well be that this person really doesn't belong in this class. They don't have the prerequisite skills that the other 98% of the class has. And so therefore they really don't even belong in this class. So that just means you have to get them in a different class or help them in some way to get caught up or, and it's gonna take more time and effort. Special causes take more time and effort. That's why they're special. So.

0:18:31.3 AS: Well, David, on behalf of everyone at The Deming Institute, I want to thank you again for this discussion. For listeners, remember to go to deming.org to continue your journey and you can also learn more about David at langfordlearning.com. This is your host, Andrew Stotz, and I'll leave you with one of my favorite quotes from Dr. Deming, "People are entitled to joy in work."