Standardized Testing: Is it an Effective Measure of Learning?

In this episode, David and Andrew discuss the dreaded standardized tests, including how they evolved, how they're used (and not used) now, and what Deming said about them. David also offers practical tips for educators who want to move away from standardized tests in their classrooms.

TRANSCRIPTION:

Stotz: My name is Andrew Stotz, and I'll be your host as we continue our journey into the teachings of Dr. W. Edwards Deming. Today, I'm continuing my discussion with David P. Langford, who has devoted his life to applying Dr. Deming's philosophy to education, and he offers us his practical advice for implementation. Today's topic is, what would Dr. Deming say about today's standardized testing? David, take it away.

Langford: So one of the things when I started learning about the Dr. Deming's field is, there's a lot of people that know about manufacturing, a lot of people know about management, etcetera, a lot of people don't, but everybody knows about education, everybody went to school somewhere, someway, homeschooling, charter schools, whatever it might be, so everybody went to school, so everybody's sort of knowledgeable on that. And one of the things that's pretty common through that process is the emphasis over the last 30 years is of standardized test, as the way to sort of measure, not only the performance of a system, or a teacher or a student individual kind of thing, but also the success of how a nation is doing, right? Can we compare the US to Japan and Australia and Thailand? And where do we rank? And who's 14th? And all these kinds of things. So I just wanna go back a little bit about, where did this evolve? Hundreds of years ago, we didn't have standardized tests, and amazingly people survived, people got jobs, became entrepreneurs, they did all kinds of things, and they didn't have a standardized test scores to be able to rank them, to be able to do stuff.

Langford: So one of the reasons these things did evolve, was a quick and easy way that we could rank individuals or rank systems, and that was one of the things that Deming was most opposed to, ranking people, either through performance appraisals, grades, standardized tests, whatever it might be. So a lot of the purpose of why we do it, is a simple way to rank schools and try to understand, "Well, who's number one? And who's not?" And that kind of a thing. And then the second one...

Stotz: And let me add something there just 'cause I wanna make sure that this comes out clear, is that obviously Dr. Deming was against the idea of ranking, but from a sincere perspective of some leaders out there, they may say, "Well, I'm not using it to rank so much as I'm just using it to figure out where is the trouble spots that I should be focusing on."

Langford: Yeah, maybe you're not doing that, but people that are getting your scores are doing that, I can guarantee it. And then now it's gone to parents looking up scores online and comparing schools even across the street to try to see, "Oh wow, that school has higher test scores than this school, so it must be a better school." Deming had a lot of comments about standardized testing, but one of the ones I remember was, he said our education system would be significantly improved if we could get rid of standardized tests and grading as a performance measure. And it goes back to... We keep talking about the purpose of the system and the constancy of purpose of the system, and so is your constancy of purpose to get good test scores? Is that really the only thing you're trying to accomplish in a school? And if that's the case, you're gonna do all kinds of really, very dysfunctional things in that school to get test scores. Like for instance, well, one of the first things we're gonna cut out is music programs and sports programs, right? 'Cause we don't test us for those, and so we don't need those, we can cut that out, and then we spend more time on this.

Langford: And I know the schools that are now mandating 90 minutes of math a day, whether you need it or not, you're gonna get 90 minutes of math a day and just waste and all kinds of stuff, and then it causes all kinds of crazy stuff within the system. And the tests themselves have slowly started to change, and one of the things that, especially that people used to say when you'd say, "Well, we should get rid of standardized tests." People say, "Well, how do we get people into colleges then? 'Cause they won't have an SAT score and they won't be able to get into college." And I think interesting thing is that there are numbers of universities, they're now throwing those things out and just saying... I know that in California, there's universities that do not require SAT scores any longer to go to the university as a measure of how well you're gonna do.

Stotz: And would just abandoning that and say, "Well, college is open for everyone." Would that be the smart thing to do? Or is it to say, "Well, let's create a system that tries to optimize for something else."? Or does that add a huge amount of complexity for a school that doesn't have a lot of resources, let's say?

Langford: Well, you want my method or do you want... Stotz: I'm only here for you.

Langford: So I don't mind that there should be some kind of a base level, you need to have graduated from a high school, you need to have done certain things, and then all the people that meet that certain baseline that wanna go into college or wanna go on to a particular college, it just goes into a lottery. And now, let's say you have a thousand spots open, so you draw out a thousand names, all those people get to come to your university or your college for that year. Now, if you weren't chosen, you know why you weren't chosen.

Stotz: Pretty clear.

Langford: Yeah, it's pretty clear. And you could put your name in for multiple universities, and if you somehow got drawn for several universities, you could make a choice, "Which one do I really wanna go to?" And you would stop all this crazy stuff about people paying to get their kids in certain universities, and some universities are so expensive that only very rich people can get their kids in those universities and just on and on and on.

Stotz: Yeah, but it could be abused. The legislators or the people in the university would say, "Well, the good news about our student body, David, is that they're all lucky."

Langford: Yeah, that's right. But if you had randomly chosen students at your university and you were still churning out some of the top students in the world in these professions, you're really doing something. You've really optimized the system in some way. On the other hand, if all you're doing is choosing the very best people that are probably gonna be successful at any place they go, you can actually get away with a lot of dysfunctional stuff, and these students are still gonna be successful because they're just gonna overcome any barriers that are put in front of them. So there's no incentive to actually optimize a system moving forward with that.

Stotz: I guess unless you're... If your method... What you're optimizing for is "I just want the top, top, top smartest, smartest as measured by score, young people in the world or in my state, or my city, or my country that's it. That's all I'm up to, man. I'm not trying to bring education to everyone," but that's not what Cal State Long Beach was doing, when I went there, they were trying to bring education to everybody.

Langford: Yeah, so it's much different constancy of purpose of what you're trying to do. The other thing about standardized tests, and we all grew up with multiple choice tests, and I remember one time somebody asking Deming at a conference about that and he paused for a long time and he said, "What do you think about multiple choice questions?" And he said, "It's ridiculous." He said, "I would much rather you have a multiple choice questions which students have to answer: in what circumstances would A and B be correct? And in what circumstances would A and D be correct?

And what would be the system if none of these were correct?" Because that would require thinking, and responses, and deep level understanding of something. But this game is still being played, especially in K-12 to get test scores, teachers are taught to teach children to guess. If you don't know the answer, just guess, because if there's four things there, you got 25% chance that you guessed right. Well, this is ridiculous, because all trying just to get your scores up. So if you really wanna find out if somebody knows something or doesn't, you certainly don't want them just guessing, that's not gonna tell you anything. And maybe 25% of the time they guess correctly, and so you're getting scores that's not telling you anything about the system.

Stotz: I was in a senior level class in my final year in university, taught by an amazing woman who had actually lived in pre-Nazi Germany, and she had just seen communism, fascism and democracy in action, and that was the name of the course. And the final exam was a essay exam, and we went in and it was, I don't know, three hours, and when I got that exam back, it said A plus, and I just felt like I just really, really understood the material, I enjoyed writing that exam and I was proud of what I accomplished. And I can say there's no multiple choice exam that I took that I'm like, "I'm really proud of what I did there."

Langford: Yeah, the same thing. We could all look back at experiences that we had, and usually they don't involve Draconian methods of testing, and standardized tests and things like that. One of the best classes I ever took in college, and I still vividly can recall or remember the name of the class was "The Assassination of American Presidents." And we went through every attempted assassination, everything, all the way up through Kennedy, and on through and analyze Zapruder films, and I don't ever remember a test in that class. Basically he just said, "Nick, if you show up every day, you're gonna get an A. Now let's start learning about this." And he had just amazing love for that whole concept and that whole thing, and that got translated to the students, and there was a waiting list to get into that class.

Stotz: And what do you say to a listener, a teacher who's listening and says, "Yeah, that's easy to do and easy to say for a senior level class or an advanced level class where you've got 15 or 20 students. But I'm teaching a freshman class of 200 students, I don't have the time to read 200 essays," How do you respond to that where they see multiple choice and standardized as kind of a weeding out, a culling, I don't know. How do you respond to that?

Langford: Well, there's lots and lots of different methods, but I would still probably require somebody to write an essay or write an explanation and go through stuff and then bring it to class, and now pass your essay once you left, and that's the person's job is to go through it and give you feedback. And then you're gonna get it back again, and then you're gonna get a chance to correct it and take that feedback, and you can do that multiple times if you want to in the class, because now you're making everybody into a teacher as well. And we all know in a classroom, the person that learns the most is the teacher.

Stotz: I'm gonna explain in a way that I've done something and then give me some critique on it or help with that. I have the valuation master class and when my students get into the advanced levels, I say, "Look, I've taught you everything I know now," and they're valuing companies, "so how much is Apple worth?" And they've gotta go through a whole pretty structured process that I put them through, and then I tell them, "Okay, when you're done, post your results up into the group, and then I'll assign another student to review the process," review what they've done and give them feedback. And then they'll have to revise, and it could be five times before I eventually look at it and give a final review and say, "Okay, here's a one little thing that you missed." And a lot of times I'll figure out, "Oh, I didn't actually teach that part," and it's being exposed because Apple is a very complex company. But the point is that, on the one hand I've been a little bit nervous about to do it because I've also felt like in an academic environment, would they say, "Well, wait a minute, are you just pure scoring or are you as the expert and the professor giving the score," how do you respond to what I'm doing and how could I improve it also?

Langford: Yeah, you have to understand that feedback is much different than rating and ranking, right? So if I write this paper and I hand it in and you give me an A minus or, you know, I might be really happy about that, or I might be just really, totally upset about that because you're ruining my GPA at the university, and it doesn't tell me a darn thing, I have no idea. You can get an A minus in one school and it's a C minus in another school, and it's an A plus in a third school, and its exact same work in all three schools, because of what Deming talked about, the variation within a system, the variability of what goes on with that. Really what you're after is feedback though, the human brain, back to neuroscience, the human brain thrives on immediate feedback, so the faster you can get that feedback to people, the better. Now, if that's peer-reviewed feedback, and that's the fastest way you can do it with 200 students at a class, that's awesome. But I always remember my son in high school was in an advanced English class, wrote a 14-page paper, worked really hard on it in September, got it back in January, and he came home with this paper and it had a grade on it, and he said, "I don't have any idea why we even wrote this thing, I don't even remember anything about it," but I got scored and moved on.

Langford: And if that's all you're after, 14-page paper, check, been there, done that and then let's move on. But if you actually wanna give people feedback, then maybe there's the whole fallacy of a 200-person class is not workable, right?

Stotz: So, in wrapping up the discussion about standardized testing, how should somebody think about it, particularly somebody that's a victim of it or in the middle of it, or being forced to do it. What's your advice?

Langford: Well, all I know is that over the last 40 years after I left high school, basically, nobody ever asked me what my standardized test score was. [laughter] No employer ever wanted to know, nobody did, they just wanted to know, Are you capable of doing the job? Are you capable of doing something? Right? And that score wouldn't have told anybody anything, even if it was a perfect score, chances are, I wouldn't wanna hire that person 'cause all they could do is just be really good at standardized tests, I need somebody with people skills in psychology and understand statistical variation, back to Deming's System of Profound Knowledge. So I would much rather our systems were preparing people to be successful in life, rather than preparing them just to take good tests.

And it's not like you can just throw the baby out with the bath water and as a teacher just say, "Well, I'm just not gonna do that anymore. I'm just not gonna test people." But there are ways that you can continually diffuse that for students so that they see it for what it is and move forward. On the other hand, you can also use a standardized test to see as a teacher, how am I doing?

Langford: Countless times I heard Dr. Deming talk about his own classes and he said, "That's what I wanna know, is how am I doing as a teacher?" Like you said, Oh, I forgot to go over certain concepts, and so a standardized test could tell me that 80% of the class didn't get this concept, and I could look back at what I did and say, Oh, no wonder, I didn't even go over that, I didn't even discuss that. So why would I wanna penalize them because I didn't do my job?

Stotz: Right. Let me try to summarize kind of what I took away from this discussion, what you were saying was that there's an emphasis on standardized tests over the past 30 years, kind of thinking maybe how is a nation doing or that score we can use to compare how different schools are doing, how different individuals are doing, and your first thing that you mentioned was, how did we survive in the past, did people learn things in the past without standardize? Of course they did. And so it doesn't mean that this is some kind of solution, and then you said that, this is the interesting part, I felt like when you were saying that it evolved as a way to rank people in systems, but ultimately it will be misused, and that is that somebody is gonna use it for the purposes that it wasn't really meant to be used for. And so it just becomes a dangerous tool. And then we talked about is the purpose to get good test scores? Because if that's what our purposes of our education system then, Yeah, go for it. Let's get the best test score. But that's not really our purpose, we also talked about how we could get people into college, maybe you talked about the random concept that they have to meet certain minimum standards, and then a random, which I found fascinating. I think it would take a lot of guts for an institution to do that, but I do think that there's definitely some interest in that.

Stotz: And then we've kind of ended up talking about feedback and using student feedback when you have a big class or you're trying to teach something at scale and how important it is to get feedback as quickly as possible. And when you were talking, I was thinking about a young person may be more concerned about what their peer thinks about what they're writing about something than the teacher. And you actually may be able to use peer pressure in a sense that they really feel like they gotta write something good, otherwise their friend next to them is gonna go, "What's this crap?" They may get more truthful response, and then finally, you talked about ultimately, our education should be about preparing people for a successful life, is there anything else that you would add to that?

Langford: I don't, you did a really good job of summarizing that up to complex subject. And it can be really fraught with emotion and there may be people listening to us that agree that that's not how we should be judging systems, but they're caught in the system itself, so you may not be able to change the system totally but you can do something. And what is that something you can do Monday morning to sort of optimize learning for the students that you're working with?

Stotz: What a challenge and a great way to end it. You may not be able to change the situation that you're in and the system that you in but you can do something, and the beginning of that something is changing your thinking. And I think that's what we're trying to get at in our discussion. So David, on behalf of everyone at the Deming Institute, I wanna thank you again for this discussion. I'm learning a lot and I know the audience is too. For listeners, remember to go to Deming.org to continue your journey. And also, David, what's the best way that someone listening or viewing this can get in touch with you to learn more?

Langford: You can go to my website at langfordlearning.com. And if you go to langfordlearning.com/booklet, you get and download a free booklet that tells you about the services that we provide in our company, so, all Deming based.

Stotz: Fantastic. Well, this is your host Andrew Stotz, and I will leave you with one of my favorite quotes from Dr Deming, "People are entitled to joy in work," and I'm gonna add learning.

Langford: Learning.