Interpreting Educational Data

In this episode, John and Andrew discuss what "transformation" means in education. John juxtaposes two reports, conducted a decade apart, that have influenced education for the last 40 years: A Nation at Risk and the Sandia Report.

TRANSCRIPT

0:00:02.3 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we continue our journey into the teachings of Dr. W. Edwards Deming. Today, I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. The topic for today is, Do we really need to transform our education system? [chuckle] John, take it away.

 

0:00:26.7 John Dues: Andrew, it's good to be back with you. Yeah, I thought... Sort of as a jumping off point from our other conversations, I remember, I think in our first conversation, you mentioned you graduated from high school, 1983 in Cleveland area, went to a solid...

 

0:00:44.9 AS: Hudson High.

 

0:00:45.2 JD: Hudson High, good traditional public school in Northeast Ohio. And your question was, if I went back to the high school 40 years later, would it look and sound the same, would it have gotten better? Would it have gotten worse? What's going on with our schools in United States, I think was the basic question, I think... When I answered you, I said two parts, there's the question about what most people probably focus on when you think about that question about Did a school get better? Did the test scores improve or decline over time? And then there was a secondary question of, Did the school transform along the lines of the Deming philosophy? And I think that those two questions would have different answers depending on which schools you're looking at, but I thought it would be interesting to sort of think about this question, Do we really need to transform our education system through the lens of a couple reports...

 

0:01:48.5 JD: Education reports, one that's well known in our world, one that's lesser known, that took a look at the... At least the test results question in the education sector, and then build from there this idea of whether or not we need to transform our schools. One thing, there's no shortage of calls to transform or some people would use the word reform our schools, and those two words probably in and of themselves, probably have different applications, but we'll use them interchangeably as we go through that question and attempt to maybe answer that over this episode and maybe a couple additional episodes.

 

0:02:36.7 AS: I find that fascinating as I observe education around the world from my own experience outside of the US, and I look at the US, and I think about the importance of education, the role of education. There's a part of education that you could say is kind of indoctrination in the way a country educates its youth to be a certain way or to understand things a certain way, so I didn't see that part of education when I was young, but now I see every country's got their indoctrination that they do within their school system, so I see it kind of broadly, but I'm just curious, really take us through what you'd like to explain about that.

 

0:03:20.4 JD: Yeah, I think the sort of start... I think there's this quote in The New Economics where Dr. Deming says that people are asking for better schools with no clear idea how to improve education, nor even how to define improvement in education, and I think if that's... And he's saying this roughly the same time that these reports are coming out, and if that's true, I think what happens is when reports come out about the state of our education sector, it's pretty easy to get pulled this way and that. When you don't have a clear picture in your mind for what schools should look like or how to improve schools, these reports have large impacts. And so the first report is well known. It came out about the same time you were graduating from high school, in 1983 in the first Reagan administration, called A Nation at Risk. It's pretty well known in the education sector, and it's had a lot of far-reaching impact in both time and place, where even today, 40 years later, we still... Some of the roots of the various reforms that we've undergone in our sector, it's still playing a role.

 

0:04:40.6 JD: The second report is, that I'll sort of juxtapose against The Nation at Risk is a report that came out about a decade later called the Sandia Report, and I think it's really interesting just to look at those two reports and the impact or lack of impact they've had over the last 30 or 40 years in the world of education. So I think I would start with, when A Nation at Risk came out, and it was commissioned by the Reagan Administration, the National Commission on Excellence in Education is the group that released the report and one of the leading statistics that's in the report is that the SAT, the college entrance exam that high school students take demonstrates a virtually unbroken decline from 1963 to 1980, where average verbal scores fell over 50 points, and average mathematic scores dropped nearly 40 points in that roughly 20 year time period. And there's these really memorable quotes that are clearly meant to awaken the public to the state of its schools that people still remember to this day, and I'll read one. It says, "We report to the American people that while we can take justifiable pride in what our schools and colleges have historically accomplished and contributed to the United States and the well-being of its people, the educational foundations of our society are presently being eroded by a rising tide of mediocrity that threatens our very future as a nation and a people."

 

0:06:24.0 JD: You couldn't get much more of wake up type people language, it's really, really interesting. Like I said, this report over the last four decades has been that foundation or bedrock for the various federal reforms that people are probably familiar with, starting with...

 

0:06:41.0 AS: And to put it into context, that's the kind of talk that was coming out of the Reagan administration, like government's not helping and government can be a problem and we need to pull ourselves up by our bootstraps and you need to take personal responsibility, so it's very... It makes sense that that type of language was coming out of the Reagan administration.

 

0:07:05.4 JD: Yeah, and I think... So this report is floating around, there's a convening of all the governors in the United States in about 1989, and some pretty strong federal education legislation starts getting put together, it starts with the first Bush and then it ends up being passed during Bill Clinton's years called Goals 2000. And has various goals around increasing graduation rates and test scores and things like that, and then that transitions to No Child Left Behind with many people are familiar with that. Came out in the early Bush years and had a lot of impact on schools when my career was first getting started down in Atlanta, but it was federal legislation, so it covered the entire country, and then it even played a role even into the Obama years when he released the Race to the Top legislation, and that was more of a competitive grant program federally that was lots of strings attached a lot of focus on test scores, a lot of focus on teacher evaluations and principal evaluations and using test scores in those evaluations.

 

0:08:21.3 JD: And so you can see this, I think, direct linkage between A Nation at Risk, to Goals 2000, to No Child Left Behind, to Race to the Top, and even to the stuff that you see at the federal level to this day. So when a report like this comes out, it's called A Nation at Risk, the thesis of the report is right in the title. A nation is at risk because of its education sector, and so it's like... Most people say, Well, we gotta do something about this. We need to take action. There's some serious implications. And so about a decade after this report comes out, the Department of Energy sort of commissions its own report. The point of this report, as you might expect, is the department of energy, they're actually looking to do some economic forecasting, so it's not directly about our schools, but they wanna take the same data set that The Nation at Risk authors looked at and analyze it.

 

0:09:32.3 JD: And interestingly, they entered this analysis thinking that they are going to verify the results from a Nation at Risk, but what actually happened is that on nearly every measure of achievement, the Sandia analysts found actually steady or slightly improving trends in the test data. So they were...

 

0:10:02.1 AS: And in the same test data or in new test data that was coming out?

 

0:10:04.7 JD: Same exact data. They actually didn't look just at test data, they were actually looking at graduation rates, dropout rate, college-going rates so on just about every one of those measures, it was either steadily improving or slightly improving. And so you go back to A Nation at Risk and you have this absolute decline in SAT scores from the early '60s to the early '80s, and the Sandia authors aren't disputing that, but they're looking at their analysis and they're saying, wait a second, this decline in average scores, actually doesn't mean that the high school students of the early '80s or early '90s, weren't as capable as their 1960s peers. And so then you start to think, Well, how could this be? It's really, really interesting. And what the Sandia report authors go on to say is that when they broke out the test scores and these other measures like I said, graduation rates and other things like that, they broke them out by race and socioeconomic status, class rank, gender, they found these steady or improving rates in all of these groups, and they chalked it up to this statistical phenomenon called Simpson's Paradox, and basically what that is, is when trends that appear in this aggregated data set, which is sort of A Nation At Risk analysis, that reverses when the data is separated into sub-groups, like it was in the Sandia report.

 

0:11:45.6 JD: So basically what they're saying is that there are a more diverse mix of students on any number of measure, socioeconomic status, gender, race, those types of things, class rank, that there's a more diverse mix of students taking this test, and that is what causes this sort of change in average test scores and other similar measures.

 

0:12:10.6 AS: Which I guess A Nation at Risk should have controlled for?

 

0:12:17.6 JD: At least... I think breaking the scores out in the way that the desegregating the data like Sandia did would have been an important step given that the population of test takers was very different in 1963 than it was in 1983 or 1993. So the Sandia researchers basically found these improving trends on dropout statistics, standardized tests, post-secondary studies, educational funding even, international assessment comparison, so all these different measures that... This sort of earlier report is raising serious alarm bells about. This new report is saying, Well, wait a second, if we look at this data and we drill down in a little bit different way, we get the opposite results, but hardly anybody knows about the Sandia report, and just about everybody in my sector, my age and older knows about the A Nation at Risk Report, it's cited all the time. Even to this day, I just heard someone on a podcast a week or two ago talking about A Nation at Risk.

 

0:13:24.7 AS: So I guess one of the lessons is be careful with how you handle data.

 

0:13:30.6 JD: Be very careful. I think one of the principless we use here is data has no meaning apart from its context, and this is a very good example of data taken out of context. I think one of the lessons for me is that when you look at our schools, and I think this is maybe what happened with A Nation at Risk, is that for most people, what you see in educational data that comes out of our schools depends, in large part, on what you thought about our schools before we looked. I think they kind of drew a conclusion and then they sort of found evidence to support that.

 

0:14:14.2 AS: Supposed to be the opposite way. Good research.

 

0:14:16.8 JD: Yeah, I think so. I think so, should have been an open question, and the Sandia Report had... I think maybe their eyes were a little more open or their willingness to consider alternative explanations was a little bit more because they were not inside the Education Sector, they were outsiders, they were physicists and economists in the Department of Energy, and so they didn't really have a dog in the fight. I guess you'd say.

 

0:14:41.5 AS: Well, I guess you could probably say we actually don't really know, but the assumption is because they're outside in the department of energy, they're completely neutral, but they may have had their own biases that they brought into that too, but still...

 

0:14:56.3 JD: Yeah, for sure. For sure.

 

0:14:58.2 AS: It's a great lesson on... What was it you said, data has no meaning without...

 

0:15:02.3 JD: Apart from its context. Yeah. Apart from its context. Yeah, I think that's a good example. Yeah.

 

0:15:06.6 AS: Yeah, and what it also makes me think about. One of the things that's so interesting about the stock market is that you can take a lot of data, you can analyze it and come up with your opinion, and let's just say that you're not that good at analyzing and you've missed some very key things in that data, and then you put your money down and the market will take it away from you, boom like that. Like as an immediate punishment for poor logic and reason, and I'd say that it's kind of one of the last places where that's kind of allowed and where it's kind of supposed to happen, but I think that the immediate punishment for bad logic and reason is not that common any place anymore.

 

0:15:53.6 JD: Yeah, I would agree. And the troubling thing is the, like I said, the wide-ranging implications that reports like A Nation at Risk can have even 40 years later.

 

0:16:11.9 AS: Yeah, and I guess that's another lesson from this, so first lesson is about understanding the data and being very careful of how you're interpreting that, the second one is that I like to say first to the mind wins. It's just... I have a funny story where I moved to Thailand and I didn't have a girlfriend and I lived with my best friend, and basically there was people at that time that took that circumstantial evidence and they said, Andrew is gay. Okay, that circumstantial evidence could point to that, and I didn't make any attempt to answer that question, so 20 years, 25 years later, a friend of mine was at a bar, and he said that he overheard two people talking about me, and they were talking about how I'm gay. And my friend went up and said, Well, actually do you guys know Andrew? And they're like, No, we've never met him. And he said, Well, I'm friends with him, and I can put this to rest that Andrew is in fact not gay. They refused to accept that. And I just thought, first of all, first opinions are very difficult to reverse. It takes a lot of emotional and intellectual energy for somebody to do that, and therefore that partially explains...

 

0:17:46.9 AS: Now, the second part that explains it, is that when you attach emotion to something, it also emboldens it or it makes it in your mind much more so if you think... If you ask an older person, Where were you when you heard that John F. Kennedy was shot? They know exactly where they were because that scary negative painful motion was attached to that particular event. So that's another lesson. But really, John, I wanna know. So my iPhones improve. The car I drive has improved. The TV I use is improved. Everything around me, the medical advancements have improved. Has education improved?

 

0:18:35.0 JD: Yeah, that's a great question because, What is education? I think probably in some places, and in some times it has and in other places, in other times it hasn't. And in the same place, in different times, the answer would probably be different and depend a lot on what it means to improve, going back to that original quote from Deming, What does improvement mean?

 

0:19:01.0 AS: So I'm asking a very non-specific general question, it sounds like what you're saying.

 

0:19:07.8 JD: Yeah. Well, and...

 

0:19:10.2 AS: Can I ask it in a little bit different way?

 

0:19:12.4 JD: Sure, because I was gonna say, before we move on from your story of the bar story, I think somewhere... There's a researcher named Zeynep Tulfekci, and I was listening to her on a podcast, I think she's some type of researcher. She said, I can't remember what they were talking about, maybe it was something COVID-related or something from a few years ago, and she said, "Whatever thing is that you're researching or just hearing about, go to the primary source and read the entire thing." And I wrote that down on a post it note.

 

0:19:45.7 AS: Nobody does that.

 

0:19:46.8 JD: 'Cause nobody does it. Now, in fact, I talk about being first to mind in some training or conversation or a book, I am sure that I heard or read about A Nation at Risk, and then I just repeated a few things over and over as if it was truth in fact, for probably 15 years before I went and read the thing myself, and my first impression reading it was, Whoa, this is all that's in here. I forget if it's 30 or 40 pages. There's not a lot of data in it. There are some compelling statistics like the SAT thing and some quotes that jump off the page, but I was struck when I actually read it for myself. There wasn't a lot there, certainly not enough to base 40 years of education reform work. That's for sure.

 

0:20:31.2 AS: And I think that's another lesson too, related to Dr. Deming's teaching. And let's say sometimes the Japanese were kind of famous about go to the location where the problem is coming from, get out of your office and go out. I think that Dr. Deming really highlighted the importance of valuing the workers and their inputs 'cause they know what's going on, and so that's something that I think if people aren't reading some of the basic research or originations of ideas, they're also probably not going down and checking out what's actually happening and you could find a very different story.

 

0:21:10.0 JD: Yeah, go to the Gemba, go to the factory floor, in our case, it's go to the classroom to see what's actually happening. Yeah. And you're gonna ask that question.

 

0:21:17.7 AS: So I wanna break my question then... I'm gonna break it down and make it a little bit more specific in hopes that you...

 

0:21:26.7 JD: You pin me down.

 

0:21:28.3 AS: Could answer it. The first question I have is that, If we go back 40 years, and I can remember, I had to take a French class and I wasn't particularly interested in France and French language, and I had no interest in that really at the time. And now, let's say it's 40 years later and a young kid like me has to take a French class: Have we come up with a better methodology for learning a language like, Okay, we've advanced, we've been teaching French for 40 years from that time to now, and now we know that there is a better way to acquire a language that cuts the language acquisition time from 40 hours to proficiency, or let's say, I don't know, 400 hours to proficiency to 300 hours to proficiency, this has nothing to do with education or the system of education, but: Have we come upon methodologies that can allow us to acquire knowledge any better or faster than what we did 40 years ago?

 

0:22:36.9 JD: That's a good question. I think... how would I answer that? I would say that in many areas of education there have been significant advances in the understanding of cognitive science or the application of cognitive science to improve teaching methods. In many areas, I think over the last 40 years, there have been advances, but like in other areas, whether or not those advances make it into the hands and the practices of the front line people is a different question.

 

0:23:23.6 AS: Which is separate. That's a separate point.

 

0:23:27.5 JD: When there's two things too, and let's take medicine for example. In medicine, there are a series of landmark trials that led to standard practices in medicine, so in education, I think in most areas, there's actually fewer of the landmark trials and key areas that everybody knows about.

 

0:23:52.4 AS: So I guess part of what I'm thinking about is one of the arguments I read in a great book called Future Hype, where the guy talked about how everybody hypes how things are moving so fast, but in fact, most of the progress that we've made in this world was made a long time ago. And he uses one example is jet airplanes, basically, we're flying at the same speed today as we did in 1950.

 

0:24:15.7 JD: 1950, yeah.

 

0:24:17.7 AS: There's been no advancement, and I can say flying back and forth from seeing Thailand and the US, there was a slight advancement where we had a plane that could fly from New York to Bangkok, but eventually they cancelled that because it was just too expensive and stuff, so it's like there really has been no... Maybe we hit the limit. And you could argue that when it comes to education, it should be quickly adopted if there's a new technology or a new way of acquiring knowledge, repetition or whatever that is, it's pretty quickly adopted, so it could be that we're at the... There's just so much that the human mind can take in.

 

0:24:55.2 JD: Well, yeah, I've heard that argument, and the second part too would be, to finish off that landmark trial thing is, in medicine where there have been landmark trials that it takes on average like 16 or 17 years for that landmark trial to then be sort of standard practice in practice by doctors and actual hospitals and clinics and even in that... In those sort of... Even when it hits that tipping point, that's far from majority...

 

0:25:23.3 AS: So you can tell the parents just wait 17 years.

 

0:25:28.6 JD: [laughter] And then we'll have this best practice for...

 

0:25:31.3 AS: I listened to somebody say that, We want you to make an investment in our education system, and the investment is your child. We'll do the best we can, but it's an investment, we're still learning and all that. So that brings me to the second part of the question is... And let's just say that education is mainly done through government in Thailand, in Asia, in Europe, in the US, I guess it's mainly done by government, but let's just say generally: Have we improved the way that we educate? Is there... I'm trying to ask it in a way that would be maybe a better way like... Okay. I don't know how to ask it, but I'll just say, like I said, my iPhone's improved tremendously. The camera that we're using on this, the microphone, the internet service that we're using to do this, all of these things have incrementally improved and at times made a major jump in improvement. And my question is, Has our ability to educate young people improved at the pace of other things or at a certain good pace?

 

0:26:50.2 JD: The way I would answer that is two parts, one, Have you ever heard of the Flynn effect?

 

0:26:56.9 AS: The what?

 

0:27:00.1 JD: The Flynn effect.

 

0:27:00.6 AS: No.

 

0:27:00.9 JD: Its name for the psychologist that discovered it. Flynn, F-L-Y-N-N. The fun fact is basically, this idea that IQs rose about three points per decade over the last century or so, I think I have that roughly right, in every population. So because of the modern world over the last 100 years has gotten more complex and there's sort of more to life that's like taking a standardized test. We've gotten better at that type of thinking over the last 100 years, so IQ has risen. So in that respect, we have gotten better, I guess, at least measures that purport to approximate whatever intelligence is. However, I don't think that we've closed gaps between groups. Those gaps that exist between different groups, performance wise, I think those... And that's sort of a key area of work for education reform movement that came out of A Nation at Risk. One of the things that people are working on is closing the achievement gaps between different groups, especially kids that are living in poverty, and their more affluent peers. I think those gaps, I think over time have been stubborn, because if you consider the Flynn effect, and that's not what's being measured on state exams, but when one group is going up and the other group is going up too, right.

 

0:28:45.7 JD: So both are relatively higher than, let's say, IQ scores were 50 years ago, but there's still this gap between groups. So again, it depends on exactly what you're talking about and determined what happened.

 

0:29:02.4 AS: And when I look at Asia, knowing the education system in Asia, first of all, over the last, let's say, 20 to 30 years, you have many, many families that have finally gotten their first kids into college, and you could argue that that's real advancement for that particular family and maybe for that society. The second thing is, you can see the culture in Asia still remains that education is very important, and so there's pressure from family and all of that in society, that it still is there, so whether American education is declining or improving, also you have to think of it in context of what's happening globally. And I think there's two ways to think about it. First is the quality of a country, ultimately the education of the people should have some effect on the quality of the country and the quality of life in that country. And then the second thing is that the position of that country in a global context should have some relationship to the level of education of that country. Those are just my ideas, it's not necessary something proven, but I feel like that could be true. So I wanna wrap this up a little bit, but how would you summarize what you want people to take away from this?

 

0:30:37.9 JD: Well, a lot of this stuff, there's sort of two counter-intuitive ideas here. When you look at these two reports that we were talking about, so on one hand, I don't think there's clear evidence that schools have been on a steady decline for the last, let's say 50 or 60 years going back to that, the early '60s that a Nation at Risk is talking about. However, on the other hand, I think that to achieve equitable outcomes for all students, that schools must undergo this transformation on an order of magnitude that's never really been seen or seldom seen in the history of organizations. And I think both reports are mostly looking at test scores and that's a pretty narrow definition of success, or there may be some uses there because we don't know how groups are doing and maybe where to allocate resources without some of those results, but they're definitely more of an inspection and in sorting mechanism than they are an improvement tool. So I think the other problem is, is that if there's this narrative that the nation is at risk, and then... Well, then you... You're saying that things are on the decline and then, Who do you blame for that decline?

 

0:32:12.9 JD: And I think what happened a lot in the last 40 years of the educational reform movement, deliberately non-deliberately, what happened was a lot of a brunt of that blame was placed on teachers and principals, the people that are working in schools. By the time Race to the Top comes out, using student test results in teacher and principal evaluations is sort of like part of getting the money that was there available through Race to the Top. And so I think my whole point with these types of reports is that, something like the Sandia report can have useful insights that maybe can facilitate some sound database decision-making, but so many times these reports come with these preconceived notions, political agendas, those types of things and the only way to make...

 

0:33:11.8 JD: To have a sound decision-making is if our education system sits on this solid philosophical foundation, and that's where I think Deming comes in, because if you have that foundation, you're not gonna make changes simply because of changes in test scores, you're gonna make changes based on whether or not something is principled and need to change according to the philosophy, and that's where I really see Deming coming in as this solid philosophical foundation, so it doesn't allow you to get swayed by a political agenda, it's a foundation that's grounded in principles, and so that's what I was thinking, we talk about in the next episode is: When you don't have those principles, what are some of the myths that emerge? And then when you identify those myths and can set those to the side, what are the principles that come in that then drive that transformation going forward. And I think Deming's work is at the center of that.

 

0:34:16.3 AS: And one of the things that makes me think about is: Can the system transform itself? And one of the ways to try to answer this question, it could be right, it could be wrong is, Is there an alternative solution for educating young people? And if there is, has there been an increase or decrease in people turning to that alternative? You could imagine that if there was a competing system and there was a huge outflow of people from one to another, then parents may say, Well, yeah, you guys can't measure what it is that is great output, but I can. That my student has homework that my child is learning, that my child is... Whatever their assessment is, and so there's someone outside, you could say the customer or the outside interested party just says, I vote with my feet. And I'm just curious, as we wrap up, Is there any knowledge that you have on what... Is there an alternative for government education in America? And has that been more or less popular over the last I don't know 10, 20 years?

 

0:35:31.0 JD: Yeah. Well, I'm gonna say the first part of your answer, I think your hypothesis, your instinct is right, is that you focused on the system. It's that focus on the system versus the focus on the individual, solely on the individuals within the system, like what was happening with the teacher and the principal evaluations and using the test data in those evaluations. So I think Deming said something like: He estimated that 94% of the problems in organization was due to the system, 6% special, and he meant 6% was maybe attributed to issues at the individual level. So the vast majority of the potential for improvement lies with the system. So I think that's what we're talking about here, the redesign of the system.

 

0:36:17.5 AS: And that also goes back to constancy of purpose, it also goes back to leadership. And is it possible that the system simply can't have constancy of purpose for political reasons or other reasons, and that... It's just a question I've never even thought about, but it is a challenge to think about, Is there constancy of purpose? Is there strong leadership without leadership...

 

0:36:45.5 JD: Yeah. There has to be fortitude there. Intestinal fortitude for sure. A strong leadership is a prerequisite. One of the things that Deming railed against was the transition, the frequent transitions amongst management leadership in the United States, because you do need that stability of leadership to maintain that focus on the aim that's guiding the system. So I think that is... That's sort of a part of the formula for success, for sure.

 

0:37:13.0 AS: I kind of interrupted you and you're, I think may be attempting to answer the question, Is there an alternative and has it grown or contracted?

 

0:37:22.0 JD: Well, so there's government-funded schools, that's traditional public schools, certainly where I am sitting in public charters, that's a government-funded school that has a slightly different governance structure. So that sector didn't exist 35 years ago, and so that now is maybe six or seven percent of the kids in the United States, something like that, attend a public charter school, and then the other component would be kids that attend a private school or are home schooled, now both of those, as I understand it, both of those populations of students rose sort of coming out of the pandemic, for sure. Yeah.

 

0:38:03.3 AS: Well, an interesting topic, and the original question is, Do we really need to transform our education system and maybe before... As we wrap up here. How would you answer that?

 

0:38:18.7 JD: So, yes, I think yes, but it's not for the reasons outlined in A Nation at Risk.

 

0:38:27.7 AS: Got it. John, on behalf of everyone at The Deming Institute, I wanna thank you again for this discussion. For listeners remember to go to Deming.org to continue your journey. This is your host Andrew Stotz, and I'll leave you with one of my favorite quotes from Dr. Deming, "People are entitled to joy in work."