The Problems With Merit Pay in Education

How do we motivate employees? Traditionally, we offer merit pay, focus on accountability, and use other extrinsic motivators tied to performance. The ideas sound good on the surface, but John and Andrew discuss the many pitfalls and unintended consequences - and what to do instead.

0:00:02.4 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we continue our journey into the teachings of Dr. W Edwards Deming. Today, I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. The topic for today is...well, in fact, we are continuing our discussions about management myths that keep fooling us. And today, we're talking about merit pay, accountability, and extrinsic motivators. John, take it away.

 

0:00:37.7 John Dues: Yeah, thanks for having me back, Andrew. We've sort of been on this sort of mini-series talking about some of these myths that Dr. Deming talked about. So two episodes ago we sort of introduced the idea that Deming said we're living in this sort of age of mythology. We talked about two myths: the myth of best practices, and the myth of the hero educator. And then last time we talked about the myth of performance appraisal, and really talked about sort of this failure to consider the role of the system on individual performance. And really what we're gonna do is kind of wrap up those myths with the three you mentioned today. I think when I think of the myth, I'm thinking about sort of management dos and don'ts, and the myths are the don'ts. And then sort of the idea would be after we cover the myths, we can turn to some guiding principles, and those would be sort of the dos, the things that management leaders should do, that sort of Dr. Deming talked about.

 

0:01:38.7 JD: So we can dive into the first one, which is sort of a continuation of last time, this idea of rating and ranking. Last time we talked about performance appraisal, and now it's sort of the merit pay side of rating and ranking. And it something...merit pay is a practice that has been sort of tried over time in education. What I can tell, it goes...the idea in education goes back at least to the Reagan administration. So at least to the '80s. So since that time, this sort of merit pay idea for teachers or other educators in the system has been taken up by various governors and presidents in the United States. I think most recently during President Obama's administration, there was the Race to the Top program. And sort of as a part of that program, there were teacher and principal evaluations where merit pay was sort of a key part of those evaluation systems.

 

0:02:36.1 JD: And so the basic theory is that if you pay people based on results, that motivator to make money will drive improvement of outcomes in schools. And I think one sort of key differentiation, because pay in all sectors, but especially in education, is a hot button topic. I'm not talking about sort of the core salary, whether or not teachers are paid enough or not. Those sort of base salary levels are sort of a separate discussion topic altogether. I'm just talking about sort of merit pay, bonus pay, performance pay, that type of thing. And when I think of merit pay, I mean a lot of these ideas, I think, sort of sound good before you really dig in and start to think about them.

 

0:03:34.0 AS: And that's, Dr. Deming would say pay for performance makes sense. Seems like you're gonna get a good outcome, but in fact it's a little different or a little bit more complex than that.

 

0:03:44.9 JD: It's a lot, yeah. A lot more complex, I think that's exactly right. When I was thinking about merit pay and the theory behind it, what are the problems, especially in education? So for me, problem number one is paramount, and that's how do you define a meaningful measure of performance by which to judge individual educators? That's a pretty thorny problem. I think problem two is that the basic theory suggests that additional money will incentivize these...improves teaching and in turn improve student outcomes. But for that to be true, that means that teachers were previously withholding their best efforts and if you just paid them this bonus, that they would then sort of unleash this previously withheld power. And then another really thorny problem is this idea, if you sort of create this environment where you have a merit pay system, it sort of disincentivizes the behaviors that are important to improving any complex system.

 

0:05:00.2 JD: Things like, cooperation and teamwork. And so, especially in a merit pay system, the ones that I've sort of been aware of, the merit pay calculation is often sort of viewed as sort of opaque. How is this calculated? Often sort of the algorithms are proprietary, they're viewed as unfair, and then they can lead to these undesirable behaviors like unwillingness to share ideas or, just as problematic, unwillingness to take on certain teaching assignments, the tougher assignments. The very kids that you want to have the best teachers are often the toughest to get results with. And so you're sort of disincentivizing people taking those assignments because of the differential pay or the poor rating.

 

0:05:57.9 AS: Yeah, it's interesting too - that the point that people are withholding their best work. If we just give them merit pay, then...and I just had a vision in my mind. Imagine that we had a peaceful cage of tigers, and they're all chilling out and these tigers consume, at every meal, they consume, let's say all of them consume a hundred kilos of meat. [laughter] And we end up putting in 50 kilos in there and say, "okay, you gotta fight for this."

 

0:06:44.9 JD: Yeah.

 

0:06:47.4 AS: What's it gonna look like?

 

0:06:48.8 JD: What's it gonna look like? Yeah.

 

0:06:50.2 AS: And in a way, like what we're doing with merit pay is saying there's a limited pool. We've allocated a pool that's available to you. Yes, you've got your survival pay, but here is this pool of additional merit pay. And then...yeah, some people, some of them may just some people may sit back and go, "I'm not doing the work for that. I'll stick with my monthly pay." Whereas others will be extremely competitive to get that pay.

 

0:07:19.8 JD: Yeah. Yeah. But all of it assumes that, let's say, today I don't have the merit pay system. I get certain results. Tomorrow, I have this merit pay system. It assumes I'd know what to do tomorrow to get the outcomes, right? And I think that's a huge problematic assumption.

 

0:07:42.7 AS: So is that's part... If you think about what you're saying, that's part of the myth that, you know, you just think...

 

0:07:47.8 JD: It's part of the the myth. Yeah.

 

0:07:49.3 AS: It's like "just put this in place." I mean, come on. It's internal competition. We want everybody to...well, wait a minute. Internal competition in the company?

 

0:08:00.4 JD: Yeah. I mean, I could work, potentially, work harder for some period of time, but if I don't have any different methods for bringing about these outcomes, then the merit pay itself is...assuming everything else went well, assuming it was seen as fair and transparent, the fact that you don't have any new methods with which to bring about this improvement is a serious impediment to just thinking that merit pay in and of itself is gonna be an effective system. Putting aside all the other issues like disincentivizing the very behaviors or taking on the various assignments that you want the sort of top-notch teachers to do.

 

0:08:36.5 AS: Yeah. I guess that's another way of thinking about it too, is that it's like you have a certain set of tools on the tool floor, on the factory floor or in the classroom or whatever. There's a certain set, and you're not adding to any of that. You're just saying, "we're gonna pay you to get more out of that," and it's just...there's a limit as to what you can get on that. All right.

 

0:09:01.9 JD: Yeah, and I think the most typical way this has showed up in, say, the last 15 years or so, are these value-added models, that instead of just focusing on the absolute test scores of individual students, what percentage are hitting that proficiency standard? The value-added models did allow you to sort of attempt to measure the progress of individual students. So even if they didn't hit proficiency, you could look at, well, did they grow a lot? And a lot of the sort of merit pay schemes, during the Race to the Top era, were based on these value-added models. But I was...just as an example, I was reading a working paper from a Cal Berkeley economics professor who looked at some of these models. And one of the things that he said really stood out, he said, Teachers gain... "Value-added model scores are evidently inflated or depressed, in part due to the students who they teach, who differ in unobserved ways that are stable over time.

 

0:10:04.5 JD: This bias accounts for as much as one third of the variation in teachers value-added scores, enough to create a great deal of misclassification in value-added model based evaluations of teacher effectiveness." So I think that that type of finding is exactly the thing that I'm talking about. Creating these models is very, very difficult. And according to at least that one research study, up to a third of the variance in the results of teachers value-added gains for their students was not attributable to the teacher themselves. It was for...to these other things.

 

0:10:41.5 AS: And so... Go ahead.

 

0:10:43.7 JD: Oh, go ahead. Yeah.

 

0:10:45.1 AS: I was just gonna say that one of the takeaways that people take from this type of discussion is, oh, I see, but okay, so merit pay needs to be better implemented.

 

[laughter]

 

0:10:56.0 AS: Right? Like, and I had...there was a LinkedIn discussion where someone posted something about KPIs, and I said the damage caused by KPIs is almost immeasurable. And I mentioned something about that and pretty much every post said it's not KPIs that are bad, it's just that we didn't train people well or they weren't explained well or they weren't implemented well. And that's a cover that can keep you doing merit pay or that type of thing for a decade - trying to create something that's fair and all of that. And I think that is part of the myth, part of the...where people get lost for years.

 

0:11:39.5 JD: Yeah. Yeah. And even in those systems, they were heavily focused on reading and math, because those were the heavy focus areas of No Child Left Behind, which was the key legislation at that time. Race to the Top was sort of a supplemental sort of grant-making process that sort of layered on top of No Child Left Behind. So even the models in reading and math weren't great at pinpointing where results were coming from. But then you also had this whole other problem where reading and math teachers make up a fairly small percentage of any staff. I mean, there's social studies teachers, science teachers, teachers in the arts, the physical education teacher, the administrators, the support staff and so you really have to finagle just the reading and math scores, to sort of make them applicable to all those other people, that have much less of a sort of direct impact on reading and math score. So that's just a sort of another problem with those systems, is how do you include the vast majority of school staff?

 

0:12:41.6 AS: And so I would say, let's wrap up this particular one by also saying that it's not about doing merit pay better.

 

0:12:50.9 JD: No, it's not. Nope. I think the practices themselves, whether it's performance appraisal or merit pay, they lead to sub optimization of the system as a whole. But I think what happens when you don't have profound knowledge, and this definitely happened to me with all of these myths, and I sort of latched onto them, when you don't have profound knowledge, these practices are continually recycled by education policy and political leaders, which is why I think you see them in the '80s during the Reagan administration, then you see 'em about 20 years later [chuckle] with the Obama administration. They get recycled, these bad ideas. When you don't have that solid philosophical foundation, you get latching onto these sort of policy implications or policies that have been tried before. You sort of forget that they didn't work the first time.

 

0:13:44.4 AS: Yeah, I think about Dr. Deming saying, how could they know?

 

0:13:47.8 JD: Right.

 

0:13:48.7 AS: And that there's just so many people that are kind of misguided by - just because something is done, that there's actually a foundational evidence that this is really the way that we optimize. So, all right, what's....

 

0:14:08.9 JD: And that was exactly what he said when he found out that President Reagan had, or his advisors, had proposed this merit pay system for schools. He said, "the problem lies in the difficulty to define a meaningful measure of performance. The only verifiable measure is a short term count of some kind, where were the President's economic advisors? He was only doing his best." So basically he was saying to the President exactly what you said. So I think the real key here is things like joy in work, intrinsic motivation, cooperation, are key to a healthy organizational culture. And these things sort of upend that. I think what we should have what Deming is telling us to do: just work to optimize the system, rather to try to incentivize those individuals working within the systems, the system as a whole that you want to work on.

 

0:15:00.7 AS: Okay. So accountability.

 

0:15:03.9 JD: Accountability. Yeah. So when we say "accountability" in school systems, what we're typically talking about is state education department accountability systems, so basically all 50 states have some type of district and school report cards. School system gets it, individual school within the school system, get them. And they're typically based on performance metrics, like proficiency rates on standardized tests, absenteeism rates, college and career readiness indicators - which on their face seem like sort of noble things to keep track of. In my home state where I am in Ohio, that sort of system trickles down to not only individual schools, but in the teacher rating system, those ratings are applied to individual educators at many traditional public schools as well. I think when we're talking about accountability systems, if you're reading Deming, he often labeled them something like "management by objective" or "management by the numbers." But really those are all the same thing. It's some type of practice where you're focused heavily on outcomes. But I think like the merit pay, several problems with the myth of accountability.

 

0:16:26.3 JD: So one we've talked about before, but I think a key one is that too often goals for accountability and goals for improvement get conflated as if they're one and the same. But accountability goals, they're sort of like inspection. They come after the fact, they don't improve the processes that produce the defective results in the first place. So when you get these results and you haven't been sort of getting sort of local data that tells you how your practices are doing, the idea is that you're supposed to then take this once a year data and then figure out what to do when it doesn't look like you want it to. And that data is not very good at that. So those two things, improvement goals and accountability goals get conflated. The sort of second problem is we talked before about how you can react to data.

 

0:17:20.9 JD: You can take it and try to improve your system, or you can distort the system itself or distort the data that's coming from the system. So a second problem, that I saw up close when No Child Left Behind was launched in the early 2000s, I was teaching in Atlanta, the legislation comes out. And if you ask teachers, what did you experience? Very often, they're gonna tell you some version of what I experienced, where as a teacher in Atlanta, we were required to spend an inordinate amount of the day on reading and math, recess got cut from an elementary schedule. I taught in an elementary school at first, and gym and the arts, while we had them, they were for very, very short periods of time and a couple times a week. Big chunks of the day on reading and math.

 

0:18:14.4 JD: And so that's a good example of distortion of the system because the legislation focused on math and reading results, to the exclusion of science and social studies and the arts and these other things. Well, that's where the school schedule then focused. And I don't think if you asked anybody that was sort of delivering schooling in that way, that that was in the best interest of kids in terms of giving them a well-rounded educational experience. The third problem is the distortion of the data. And this is something that happened many places, especially during Bush two and Obama's administrations, when No Child Left Behind was in full swing. So if you ask any educator and even many people outside of education, can you remember some major cheating scandals that happen with state test scores? Everybody can remember a few. There's a big one in Atlanta a few years after I left. I know there was a major one in DC.

 

0:19:13.1 JD: There was a major one here in Columbus in the Columbus City schools in like the 2013-2012 range, and they happened all over the country. I mean, even the one in Atlanta, the superintendent of the schools was charged with running a corrupt organization. They used the RICO statute because they were actually giving bonuses based on test scores that were - I forget how they were cheating exactly. And obviously, this isn't the majority of people. But it is sort of a product of a system that's putting so much focus on these test scores, and then you're layering the merit pay on top of it, and this is sort of what you get.

 

0:20:01.3 AS: That's the problem around here. We don't have enough accountability.

 

0:20:04.2 JD: Right, right.

 

0:20:05.4 AS: So, we're getting everybody accountable, everybody's gonna be... And we are gonna get tough on accountability.

 

0:20:12.5 JD: Yeah, yeah.

 

0:20:13.1 AS: Squeeze.

 

0:20:14.0 JD: I think that's the Deming sort of point with management by objective accountability system, is stop holding people accountable in lieu of improving processes. Of course accountable to our teammates, Deming talks about a system, where do you fall in the system? Understanding the system's view versus the organizational chart. Who's relying on me? Who do I rely on? Those are important...that's accountability. But what he's talking about is when all of this focuses on accountability by inspection rather than sort of working together to improve the processes that ultimately lead to the outcome.

 

0:20:54.1 AS: Yeah, and seeing the data as a tool, a feedback mechanism that helps us understand, and test what we're doing with the system. So, yeah.

 

0:21:05.2 JD: I think another thing that's really underappreciated is that one, numerical goals don't produce quality especially when those goals are outside the capability of the system as it's currently designed. So, if you remember back when we looked at those third grade reading test scores, they were sort of bouncing around about a 60% average, if I remember right. They were like 58% then up to 62% and down to 60% and up to 61% and down to 59%. And so, they're bouncing around about a 60% average, the goal is 80%. That goal is outside of the capability of the system.

 

0:21:40.6 JD: And so, if people over time, depending on what sanctions are being issued, realize that there is literally almost no chance that they're gonna hit that mark within the current system, what are they gonna do? Again, they can work to improve the system, which is hopefully what happens. But you're very likely to see some type of distortion of the system or distortion of the data, even if it doesn't rise to cheating on state tests. I only report out on certain data results that make my organization look good or whatever. I spend all my time trying to write this fiction instead of actually improving the system. And I think that's something that happens all the time, and we don't understand what's the current system capable of.

 

0:22:23.7 AS: And how do you counter the argument that some people say is that: some of your employees, some of your teachers or administrators, just they don't care that much, and they need accountability. If you don't have accountability, they're not gonna step up and try to improve the system and all that stuff. How do you handle somebody who says, "You're living in a fantasy world, John. And the fact is that we need to crack the whip around here." [laughter]

 

0:22:50.4 JD: Yeah. Yeah. I can appreciate that, I think I grew up with, show up on time and do your job. That was a part of my upbringing. However, it just hasn't been my experience that most people are slackers. Sure, have there been a handful of people across my 20 years in various cities and states that probably shouldn't be teaching or leading a school? Yeah. I've definitely come across some people like that.

 

0:23:18.1 AS: And I guess the answer to that is really, that's a management job, to assess a person and try to make a decision, coach them, help them, move them, whatever.

 

0:23:27.1 JD: It's a management decision or a management sort of responsibility. And I think a key thing is, when I have seen teachers that maybe would have the attitude that you're describing. If you unpack why that is, it's often in education because let's say there are a 15-year teacher... I saw this a little bit in Atlanta. I was a newer teacher, there were some 15-year teachers on my team. They were gonna do their own thing, but if you unpack why that is, it's because, "look, I've been through four or five superintendents in my 15 years, each one brought a new set of reform ideas. I had to get trained on all these new things, and then those things were gone in two or three years. And I've been through that cycle three or four or five times." Of course, you're gonna have that attitude, who's not gonna have that attitude? You sort of have initiative fatigue.

 

0:24:18.4 AS: Yeah. And I'm sure as those cycles went through, there was times that those people were beaten down on a particular thing that they didn't get right, and then it's like, "Sorry, I'm not taking the risk."

 

0:24:30.3 JD: Yeah, yeah.

 

0:24:31.1 AS: Alright.

 

0:24:31.3 JD: So that just goes back to, the problematic employee is a very small percentage. And generally what we're talking about are systems problems that require leadership to fix. Yeah.

 

0:24:44.3 AS: So: extrinsic motivators.

 

0:24:48.7 JD: Yeah. I think the things we've been talking about are extrinsic motivators too. Performance pay, performance appraisal systems, those are all sort of forms of extrinsic motivators. But I think the basic premise of this myth is that you can improve performance by putting the right extrinsic motivators into place. So, the basic theory or supposition is that if you just get the right balance between reward and punishment, then that's gonna improve your system. And I think it's definitely true that people are differentialy motivated by extrinsic and intrinsic factors.

 

0:25:31.9 JD: Some people require far less of the other, or one or the other than other people. But I think it is a false premise that you can improve performance using carrots and sticks. Just sort of full stop. And I think what I've seen when I've read about this is that, one, in most contexts within complex systems like schools, extrinsic motivators don't work to improve performance like some people might think they do. They typically only work in the short term, and even in those short term settings, they typically only work for simple and repetitive tasks. And those don't sound like what teachers or principals do on a daily basis, basically.

 

0:26:24.6 JD: I think there's also these unintended consequences that stem from practices like teacher evaluations and merit pay that heavily rely on extrinsic motivation, because they do lead to these distortions in the data or the data within the system. Distortions of the system or data within the system. Because again, they're optimizing competition within the organization versus cooperation. They actually make it harder to achieve your goals. And then I think that this all brings us to the primary issue when you try to use extrinsic motivators targeted at individuals, is that individual performance only accounts for a tiny fraction of organizational performance.

 

0:27:18.2 JD: And I think Deming pegged that number depending on the exact situation, that 94% to 97% of the troubles and possibilities for improvement in any given organization actually belong to the system, and are the responsibility of management. So, if you do the opposite and try to incentivize individuals who have little control over the system in the first place, at best it's pointless. And at worst, these incentive systems have the exact opposite effect and actually decrease organizational performance. [laughter] I think that's really what is behind the Deming philosophy. The common thread is that these things work to sub-optimize the system as a whole, all of them. So yeah, when you fail to appreciate the organization as a system, you're actually making the improvement of that system much, much, much harder. The Deming philosophy does the opposite of that.

 

0:28:20.4 AS: Yeah. I'm thinking about taking like a test tube and putting some things in it and then shaking it up, [laughter] agitating it, and really putting pressure on it and all that. And then when you open the container, it goes, pop, and it explodes because you've agitated. And you're agitating a small part of the overall, that's a part that has probably the smallest impact on the overall system, of the overall output of the system. So, yeah. Interesting. [laughter]

 

0:29:00.0 JD: Well, and I think a key part of this mentally is that even for people that have at least a moderate grasp of the Deming philosophy, when you're actually a leader and you're in a high pressure situation: I just got my state test scores, well, how did we do last year? We have to have done better than last year, and we have to do better than these three competitors. As long as that's okay, I'm okay, I'm okay. But then you sort of have forgotten in the moment of stress all of the things you've learned about this philosophy. Like, well, how does this data look over time? Is it really just bouncing around? It's not really improvement. But the trick is assuming that you know the philosophy in the first place, that you can then step back and employ it or fully use it in those times of stress when community members or board members, your boss, is asking you for these results and you're trying to figure out a way to sort of paint them in a positive light.

 

0:30:00.0 JD: And so, even if you're a believer in this, you can see how people start to distort that system or distort the data. Again, not rising to the level of the cheating scandals that I talked about in Atlanta and Columbus, but in these everyday and in these everyday ways. So, I think that's why you have to explicitly say the Deming philosophy - it's how you manage your organization. It's why you have to get top leaders an understanding of the philosophy. And then you have to live it day in and day out, not only when it's easy, but especially in those times of high pressure when you wanna revert back to the myths.

 

0:30:42.9 AS: And the exam results come out.

 

0:30:43.8 JD: When the exam results come out, and someone's calling, you have to be ready to talk about it in this way, even in those high pressure, those pressure time periods, yeah. Yeah.

 

0:30:53.6 AS: So, let's wrap this up. We're talking about the management myths that keep fooling us, and we're talking about merit pay, accountability, and extrinsic motivators. What we were saying about merit pay, you were saying that it assumes that additional money is what's going to get a better output from teachers as if maybe they're holding something back right now. And also, you mentioned that it disincentivizes teamwork, and it's not clear how it's calculated, and it leads to sub-optimization of the system. About accountability, you were talking about how Dr. Deming talked about management by objective or management by numbers and the problems that you face with that. You also talked about the accountability versus improvement goals. I think that was a really helpful discussion to kind of understand that improvement and accountability are two separate things. And then also, you talked about distortions of the data, distortions of the systems and that type of things.

 

0:32:01.5 AS: And that only also you mentioned finally on the accountability is that, if you're holding people accountable to numerical goals that are outside of the system's capability, it's not reasonable at all. And then extrinsic motivators, you talked about that there's a myth that we can improve performance with the right balance between reward and punishment. But you mentioned about unintended consequences of that and causing distortions of the system. And in a sense, you're optimizing for competition within the system rather than cooperation and coordination. And then finally you wrap that up, which probably applies to all of it, with the idea that Dr. Deming clearly stated that the output of a system, maybe, 95%, 96% or so of the output of a system is actually attributable to the system, not the individuals running about doing the best that they can within that system. And so therefore, it doesn't make so much sense to overly focus on these things when they really are not the key to getting the output that you want. Is there anything you would add to that?

 

0:33:18.6 JD: Yeah, no, that's exactly right. I think systems leaders have to understand the management myths, so they can avoid them. And then sort of the next step beyond avoiding the management myths is, well, what do you do? What are the things that you do? And that's informed by the system of profound knowledge. And I think it's also informed by that set of guiding principles that I alluded to at the beginning. And I think that's sort of where to go next. Those are the dos of the Deming management philosophy. So, you have some principles to operate by, so you don't get caught up in these myths when things get tough.

 

0:33:54.3 AS: Beautiful. Well, John, on behalf of everyone at the Deming Institute, I wanna thank you again for this discussion. For listeners, remember to go to deming.org to continue your journey. This is your host, Andrew Stotz saying, I'll leave you with one of my favorite quotes from Dr. Deming. "People are entitled to joy in work."