In Their Own Words

Jacob Stoller, a Shingo prize-winning author and expert on Lean management, delves into the myths surrounding traditional productivity. He shares his evolution from sales to Lean journalism, emphasizing the human element in Lean practices. The conversation highlights the resistance organizations face when adopting these principles, particularly from management. Stoller challenges misconceptions that resource cuts equal better profits, advocating for a deeper understanding of interdependence and continuous improvement in productivity strategies.

John Dues, an innovative educator passionate about applying Dr. Deming's principles, shares his insights on building an effective improvement model for schools. He discusses the importance of understanding system capabilities before setting goals and advocates for collaboration in achieving ambitious objectives. The conversation dives into navigating uncertainty in goal setting and enhancing student attendance through structured approaches. John emphasizes the power of data analysis and iterative processes to foster continual improvement in education.

Bill Bellows, a quality management expert with over 31 years of experience, and Andrew Stotz, devoted to Dr. W. Edwards Deming’s teachings, dive into the world of quality. They explore whether striving for A+ quality is always beneficial, dissecting the concepts of acceptability versus desirability. Listeners will learn about different quality philosophies, such as zero defects and Six Sigma, as well as the importance of aligning quality goals with financial viability and organizational objectives.

Quality expert Bill Bellows shares insights on Dr. Deming's teachings and David Garvin's 8 Dimensions of Quality. They discuss the importance of understanding and managing variation, aggressive quality strategies for market success, and exploring the critical dimensions of quality like performance, features, and serviceability.

Bill Bellows, with 31 years of experience in Dr. Deming's philosophy, shares his quality journey. Topics include challenges in understanding quality management teachings, first encounters with quality circles, Taguchi method for gear wear problems, and Deming's philosophies on quality and systems thinking.

In the final episode of the goal setting in classrooms series, John Dues and Andrew Stotz discuss the last three of the 10 Key Lessons for implementing Deming in schools. They finish up with the example of Jessica's 4th-grade science class. TRANSCRIPT 0:00:02.4 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we continue our journey into the teachings of Dr. W Edwards Deming. Today I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. This is episode six about goal setting through a Deming lens. John, take it away.   0:00:26.4 John Dues: Hey, Andrew, it's good to be back. Yeah, for the past handful of episodes or so, we've been talking about organizational goal setting. We covered these four conditions of healthy goal setting and then got into these 10 key lessons for data analysis. And then we've been looking at those 10 key lessons applied to an improvement project. And we've been talking about a project that was completed by Jessica Cutler and she did a Continual Improvement Fellowship with us here at our schools. And if you remember, Jessica was attempting to improve the joy in learning of her students in her fourth grade science class. So last time we looked at lessons five through seven. Today we're gonna look at those final three lessons, eight, nine and ten applied to her project.   0:01:15.7 AS: It's exciting.   0:01:17.1 JD: Yeah. So we'll jump in here. We'll kind of do a description, a refresher of each lesson. And we'll kind of talk about how it was applied to her specific project, and we'll look at some of her data to kind of bring that live for those of the folks that have video. Let's jump in with lesson number eight. So we've talked about this before, but lesson number eight was: more timely data is better for improvement purposes. So we've talked about this a lot. We've talked about something like state testing data. We've said, it can be useful, but it's not super useful for improvement purposes, because we don't get it until the year ends. And students in our case, have already gone on summer vacation by the time that data comes in. And you know that the analogous data probably happens in lots of different sectors where you get data that lags, to the point that it's not really that useful for improvement purposes.   0:02:15.8 JD: So when we're trying to improve something, more frequent data is helpful because then we can sort of see if an intervention that we're trying is having an effect, the intended effect. We can learn that more quickly if we have more frequent data. And so it's, there's not a hard and fast rule, I don't think for how frequently you should be gathering data. It just sort of needs to be in sync with the improvement context. I think that's the important thing. Whether it's daily or a couple times a day or weekly, or monthly, quarterly, whatever, it's gotta be in sync with whatever you're trying to improve.   0:02:50.5 AS: You made me think about a documentary I saw about, how they do brain surgery and how the patient can't be sedated because they're asking the patient questions about, do you feel this and they're testing whether they're getting... They're trying to, let's say, get rid of a piece of a cancerous growth, and they wanna make sure that they're not getting into an area that's gonna damage their brain. And so, the feedback mechanism that they're getting through their tools and the feedback from the patient, it's horrifying to think of the whole thing.   0:03:27.7 JD: Yeah.   0:03:28.3 AS: It's a perfect example of why more timely data is useful for improvement purposes 'cause imagine if you didn't have that information, you knock the patient out, you get the cancerous growth, but who knows what you get in addition to that.   0:03:43.7 JD: Yeah, that's really interesting. I think that's certainly an extreme example, [laughter], but I think it's relevant. No matter what our context, that data allows us to understand what's going on, variation, trends, whether our system is stable, unstable, how we should go about improving. So it's not dissimilar from the doctors in that example.   0:04:06.8 AS: And it's indisputable I think, I would argue. But yet many people may not, they may be operating with data that's not timely. And so this is a reminder that we would pretty much always want that timely data. So that's lesson eight. Wow.   0:04:22.6 JD: Lesson eight. Yeah. And let's see how we can, I'll put a visualization on the screen so you can see what Jessica's data look like. All right. So now you can see. We've looked at these charts before. This is Jessica's process behavior chart for joy in science. So just to reorient, you have the joy percentage that students are feeling after a lesson on the x-axis, sorry, on the y-axis. On the x-axis, you have the school dates where they've collected this survey information from students in Jessica's class.   0:04:57.0 AS: Can you put that in Slide Show view?   0:05:00.4 JD: Yeah. I can do that. Yeah.   0:05:02.7 AS: Just it'll make it bigger, so for the...   0:05:06.5 JD: There you go.   0:05:07.8 AS: For the listeners out there, we're looking at a chart of daily, well, let's say it looks like daily data. There's probably weekends that are not in there because class is not on weekends, but it's the ups and downs of a chart that's ranging between a pretty, a relatively narrow range, and these are the scores that are coming from Jessica's surveying of the students each day, I believe. Correct?   0:05:34.2 JD: Yeah. So each day where Jessica is giving a survey to assess the joy in science that students are feeling, then she's averaging all those students together. And then the plot, the dot is the average of all the students sort of assessment of how much joy they felt in a particular science lesson.   0:05:54.7 AS: And that's the average. So for the listeners out there John's got an average line down the middle of these various data points, and then he is also got a red line above and a red line below the, above the highest point and slightly below the lowest point. Maybe you can explain that a little bit more.   0:06:15.4 JD: Yeah. So with Jessica, you remember originally she started plotting on a line chart or a run chart when we just had a few data points just to kind of get a sense of how things are moving so she could talk about it with her class. And over time what's happened is she's now got, at this point in the project, which she started in January, now this is sort of mid-March. And so she's collected two to three data points a week. So she doesn't survey the kids every day just for time sake, but she's getting two, three data points a week. And so by March, she started just a couple months ago, she's got 28 data points. So that sort of goes back to this idea of more timely data is better for improvement.   0:07:00.9 JD: And a lot of times, let's say a school district or a school does actually survey their students about how, what they think of their classes. That might happen at best once a semester or maybe once a year. And so at the end of the year you have one or two data points. So it's really hard to tell sort of what's actually going on. Compared to this, Jessica's got these 28 data points in just about two months or so of school. So she's got 28 data points to work with. And so what her and her students are doing with this data then, one, they can see how it's moving up and down. So we have, the blue dots are all the plotted points, like you said, the green line is the average running sort of through the middle of the data, and then those red lines are our process limits, the upper and lower natural process limits that sort of tell us the bounds of the system.   0:07:50.4 JD: And that's based on the difference in each successive data point. But the most important thing is that as Jessica and her students are looking at this, initially, they're really just studying it and trying to sort of see how things are going from survey to survey. So one of the things that Deming talked about frequently is not tampering with data, which would be if you sort of, you overreact to a single data point. So let's say, a couple of days in, it dips down from where it started and you say, oh my gosh, we gotta change things. And so that's what Deming is talking about. Not tampering, not overreacting to any single data point. Instead look at this whole picture that you get from these 28 data points and then talk about...   0:08:41.5 JD: In Jessica's case she's talking about with her students, what can we learn from this data? What does the variation from point to point look like? If we keep using the system, the fourth grade science system, if we leave it as is, then we'll probably just keep getting data pretty similar to this over time, unless something more substantial changes either in the negative or the positive. So right now they...   0:09:10.1 AS: And I think for the listeners, it's, you can see that there's really no strong pattern that I can see from this. It's just, there's some, sometimes that there's, seems like there's little trends and stuff like that. But I would say that the level of joy in the science classroom is pretty stable.   0:09:32.1 JD: Pretty stable. Yeah. Pretty high. It's bouncing around maybe a 76% average across those two and a half months or so. And so, they, you kind of consider this like the baseline. They've got a good solid baseline understanding of what joy looks like in this fourth grade science classroom. Did that stop sharing on your end?   0:10:00.2 AS: Yep.   0:10:00.2 JD: Okay, great. So that's lesson eight. So clearly she's gathered a lot of data in a pretty short amount of time. It's timely, it's useful, it's usable, it can be studied by her and her students. So we'll switch it to lesson nine now. So now they've got a good amount of data. They got 28 data points. That's plenty of data to work with. So lesson nine is now we wanna clearly label the start date for an intervention directly in her chart. And remember from earlier episodes, not only are we collecting this data, we're actually putting this up on a screen on a smart board in the classroom, and Jessica and her students are studying this data together. They're actually looking at this, this exact chart and she's explaining sort of kind of like we just did to the listeners. She's explaining what the chart means.   0:10:54.2 JD: And so over time, like once a week she's putting this up on the smart board and now kids are getting used to, how do you read this data? What does this mean? What are all these dots? What do these numbers mean? What do these red lines mean? That type of thing. And so now that they've got enough data, now we can start talking about interventions. That's really what lesson nine is about. And the point here is that you want to clearly, explicitly with a literally like a dotted line in the chart to mark on the day that you're gonna try something new. So you insert this dashed vertical line, we'll take a look at it in a second, on the date the intervention started. And then we're also gonna probably label it something simple so we can remember what intervention we tried at that point in time.   0:11:42.7 JD: So what this then allows the team to do is then to very easily see the data that happened before the intervention and the data that happened after the implementation of this intervention or this change idea. And then once we've started this change and we start plotting points after the change has gone into effect, then we can start seeing or start looking for those patterns in the data that we've talked about, those different rules, those three rules that we've talked about across these episodes. And just to refresh, rule one would be if we see a single data point outside of either of the limits, rule two is if we see eight consecutive points on either side of that green average line, and rule three is if we see three out of four dots in a row that are closer to one of the limits than they are to that central line.   0:12:38.3 JD: So that again, those patterns tell us that something significant, mathematically improbable has happened. It's a big enough magnitude in change that you wouldn't have expected it otherwise. And when we see that pattern, we can be reasonably assured that that intervention that we've tried has worked.   0:12:56.0 AS: And let me ask you about the intervention for just a second because I could imagine that if this project was going on, first question is, does Jessica's students are, obviously know that this experiment is going on?   0:13:08.3 JD: Yes.   0:13:09.8 AS: Because they're filling out a survey. And my first question is, do they know that there's an intervention happening? I would expect that it would be yes, because they're gonna feel or see that intervention. Correct?   0:13:25.1 JD: Sure. Yep.   0:13:25.2 AS: That's my first point that I want to think about. And the second point is, let's imagine now that everybody in the classroom has been seeing this chart and they're, everybody's excited and they got a lot of ideas about how they could improve. Jessica probably has a lot of ideas. So the temptation is to say, let's change these three things and see what happens.   0:13:46.5 JD: Yeah.   0:13:47.1 AS: Is it important that we only do one thing at a time or that one intervention at a time or not? So maybe those are two questions I have in my mind.   0:13:58.6 JD: Yeah, so to the first question, are you, you're saying there there might be some type of participant or...   0:14:02.3 AS: Bias.   0:14:03.3 JD: Observer effect like that they want this to happen. That's certainly possible. But speaking to the second question, what intervention do you go with? Do you go with one or you go with multiple? If you remember a couple of episodes ago we talked about, and we actually looked at a fishbone diagram that Jessica and her students that they created and they said, okay, what causes us to have low joy in class? And then they sort of mapped those, they categorized them, and there were different things like technology not working. If you remember, one was like distractions, like other teachers walk into the room during the lesson. And one of them was others like classmates making a lot of noise, making noises during class and distracting me. And so they mapped out different causes. I think they probably came up with like 12 or 15 different causes as possibilities.   0:14:58.7 JD: And they actually voted as a class. Which of these, if we worked on one of these, which would have the biggest impact? So not every kid voted for it, but the majority or the item that the most kids thought would have the biggest impact was if we could somehow stop all the noises basically. So they came up with that as a class, but not, it wasn't everybody's idea. But I think we've also talked about sort of the lessons from David Langford where once kids see that you're gonna actually take this serious, take their ideas serious and start acting on them, they take the project pretty seriously too. So maybe not a perfect answer, but that's sort of what we...   0:15:38.0 AS: I was thinking that, ultimately you could get short-term blips when you do an intervention and then it stabilizes possibly. That's one possibility. And the second thing I thought is, well, I mean ultimately the objective, whether that's an output from a factory, and keeping, improving that output or whether that's the output related to joy in the classroom as an example, you want it to go up and stay up and you want the students to see it and say, wow, look, it's happening. So, yeah.   0:16:11.7 JD: And there's different ways you can handle this. So this joy thing could go up to a certain point. They're like, I don't know if we can get any more joy, like, it's pretty high. And what you could do at that point is say, okay, I'm gonna assign a student to just sort of, every once in a while, we'll keep doing these surveys and we will sort of keep plotting the data, but we're not gonna talk about a lot. I'm just gonna assign this as a student's job to plot the new data points. And we'll kind of, we'll kind of measure it, but we won't keep up with the intervention 'cause we got it to a point that we're pretty happy with. And now as a class we may wanna switch, switch our attention to something else.   0:16:45.2 JD: So we started getting into the winter months and attendance has dipped. Maybe we've been charting that and say, Hey guys, we gotta, gotta kinda work on this. This is gone below sort of a level that's really good for learning. So let's think about as a group how we could come up with some ideas to raise that. So maybe you turn your attention to something else, 'cause you can't pay attention to everything at once.   0:17:07.2 AS: Yeah, and I think I could use an example in my Valuation Master Class Boot Camp where students were asking for more personal feedback and I realized I couldn't really scale this class if I had to get stuck into hundreds of grading basically. And that's when I came up with the concept of feedback Friday, where one student from each team would present and then I would give feedback, I would give a critique and they would be intense and all students would be watching, it would be recorded, and all of a sudden all the issues related to wanting this personal feedback went away. And therefore, once I instituted it on a regular basis, I went on to the next issue and I made sure that I didn't lose the progress that I had made and continue to make feedback Friday better and better.   0:17:56.2 JD: Yeah. Yeah. That's great. That's great. I'll share my screen so you can kinda see what this looked like in Jessica's class now, what the chart looks like now. So now you see that same chart, that same process behavior chart, exact same one we were just looking at except now you can see this, this dashed vertical line that marks the spot where the intervention was started that we just talked about. And what the kids are actually doing, and Jessica are running a PDSA cycle, a Plan-Do-Study-Act cycle. That's the experimental cycle in her class. And what they're running that PDSA on is, again, how can we put something in place to reduce the distracting noises. And so what the students actually said is if we get a deduction for making noises, then there will be less noises. And so in the school's sort of management system, a deduction is sort of like a demerit.   0:19:00.0 JD: If you maybe went to a Catholic school or something like that, or some public schools had demerits as well, but basically it's like a minor infraction basically that goes home or that gets communicated to parents at the end of the week. But the kids came up with this so their basic premise is, their plan, their prediction is if there are less noises, we'll be able to enjoy science class. And if we give deductions for these noises, then there'll be less noises. So some people may push back, well, I don't think you should give deductions or something like that, but which, fine, you could have that opinion. But I think the powerful point here is this is, the students created this, it was their idea. And so they're testing that idea to see if it actually has impact.   0:19:44.8 JD: And they're learning to do that test in this scientific thinking way by using the Plan-Do-Study-Act cycle, and seeing if it actually has an impact on their data. So at the point where they draw this dashed line, let's call that March 19th, we can see a couple of additional data points have been gathered. So you can see the data went up from 3/18 to 3/21. So from March 18th to March 21st, rose from about, let's call it 73% or so, up to about 76% on March 21st. And then that next day it rose another percent or two and let's call that 78%.   0:20:28.1 JD: And so the trap here is you could say, okay, we did this intervention and it made things better. But the key point is the data did go up, but we haven't gathered enough additional data to see one of those patterns that we talked about that would say, oh, this actually has had a significant change. Because before the dashed line, you can see data points that are as high or even higher than some of these ones that we see after the PDSA is started. So it's too early to say one way or another if this intervention is having an impact. So we're not gonna overreact. You could see a place where you're so excited that it did go up a couple of days from where it was on March 18th before you started this experiment, but that's a trap. Because it's still just common cause data, still just bouncing around that average, it's still within the bounds of the red process limits that define the science system.   0:21:34.2 AS: I have an experiment going on in my latest Valuation Master Class Boot Camp, but in that case, it's a 6-week period that I'm testing, and then I see the outcome at the end of the six weeks to test whether my hypothesis was right or not. Whereas here it's real time trying to understand what's happening. So yes, you can be tempted when it's real time to try to jump to conclusion, but when you said, well, okay, I can't really get the answer to this conclusion until I've run the test in a fixed time period, then it's you don't have as much of that temptation to draw a conclusion.   0:22:14.1 JD: Yeah. And if I actually was... I should have actually taken this a step farther. I marked it with this Plan-Do-Study-Act cycle. What I should have done too is write "noises" or something like that, deduction for noises, some small annotation, so it'd be clear what this PDSA cycle is.   0:22:32.1 AS: In other words, you're saying identify the intervention by the vertical line, but also label it as to what that intervention was, which you've done before on the other chart. I remember.   0:22:42.1 JD: Yeah. And then it'd be sort of just looking at this when she puts this up on the smart board for the class to see it again too. Oh yeah yeah, that's when we ran that first intervention and that was that intervention where we did deductions for noises. But the bigger point is that this never happens where you have some data, you understand a system, you plan systematic intervention, and then you gather more data right after it to see if it's having an impact. We'd never do that ever, in education, ever. Ever have I ever seen this before. Nothing like this. Just this little setup combining the process behavior chart with the Plan-Do-Study-Act cycle, I think is very, very, very powerful and very different approach than what school improvement.   0:23:33.4 AS: Exciting.   0:23:34.6 JD: Yeah. The typical approach is to school improvement. So I'll stop that share for a second there, and we can do a quick overview of lesson 10 and then jump back into the chart as more data has been gathered. So lesson 10 is: the purpose of data analysis is insight. Seems pretty straightforward. This is one of those key teachings from Dr. Donald Wheeler who we've talked about. He taught us that the best analysis is the simplest analysis, which provides the needed insight.   0:24:08.1 AS: So repeat lesson 10, again, the purpose of...   0:24:11.6 JD: The purpose of data analysis is insight.   0:24:14.7 AS: Yep.   0:24:15.6 JD: So just plotting the dots on the run chart and turning the run chart into the process behavior chart, that's the most straightforward method for understanding how our data is performing over time. We've talked about this a lot, but it's way more intuitive to understand the data and how it's moving than if you just stored it in a table or a spreadsheet. Got to use these time sequence charts. That's so very important.   0:24:42.2 AS: And I was just looking at the definition of insight, which is a clear, deep, and sometimes sudden understanding of a complicated problem or situation.   0:24:51.6 JD: Yeah. And I think that can happen, much more likely to happen when you have the data visualized in this way than the ways that we typically visualize data in just like a table or a spreadsheet. And so in Jessica's case, we left off on March 22nd and they had done two surveys after the intervention. And so then of course what they do is they continue over the next 4, or 5, 6 weeks, gathering more of that data as they're running that intervention, then we can sort of switch back and see what that data is looking like now.   0:25:28.3 AS: Exciting.   0:25:30.3 JD: So we have this same chart with that additional data. So we have data all the way out to now April 11th. So they run this PDSA for about a month, three weeks, month, three, four weeks.   0:25:47.9 AS: And that's 11 data points after the intervention. Okay.   0:25:54.0 JD: Yep. Purposeful. So what was I gonna say? Oh, yeah. So three, four weeks for a Plan-Do-Study-Act cycle, that's a pretty good amount of time. Two to four weeks, I've kind of found is a sweet spot. Shorter than that, it's hard to get enough data back to see if your intervention has made a difference. Longer than that, then it's you're getting away from the sort of adaptability, the ability to sort of build on an early intervention, make the tweaks you need to. So that two to four week time period for your PDSA seems like a sweet spot to me. So she's continued to collect this joy in learning data to see... Basically what her and her class are doing is seeing if their theory is correct. Does this idea of giving deductions for making noises have an impact? Is it effective?   0:26:44.0 JD: So if they learn, if the data comes back and there is no change, no indication of improvement, then a lot of people will say, well, my experiment has failed. And my answer to that is, no, it hasn't failed. It might not have worked like you wanted, but you learn very quickly that that noise deduction is not going to work and we're gonna try some other thing, some other intervention. We learn that very very quickly within 3 or 4 weeks that we need to try something new. Now, in the case of Jessica's class, that's not what happened. So you can actually see that dotted line, vertical dotted line is still at March 19th, we have those 11 additional data points. And you can actually see, if you count, starting with March 21st, you count 1-2-3-4-5-6-7-8-9-10-11 data points that are above that green average line from before.   0:27:45.5 JD: So originally the red lines, the limits and the central line would just be straight across. But once I see that eight or more of those are on one side of that central line, then I actually shift the limits and the average line, 'cause I have a new system. I've shifted it up and that actually is an indication that this intervention has worked, because we said... Now for those that are watching, it doesn't appear that all the blue dots are above that green line, but they were before the shift. Remember the shift indicates a new system. So I go back to the point where the first dot of the 8 or more in a row occurred, and that's where I have indicated a new system with the shift in the limits and the central line. So this, their theory was actually correct. This idea of giving a deduction for noises actually worked to improve the joy in Jessica's science class. It was a successful experiment.   0:28:52.7 AS: Can I draw on your chart there and ask some questions?   0:29:00.5 JD: Sure. Yeah.   0:29:00.6 AS: So one of my questions is, is it possible, for instance, in the preliminary period, let's say the first 20 days or so that things were kind of stabilized and then what we saw is that things potentially improved here in the period before the intervention and that the intervention caused an increase, but it may not be as significant as it appears based upon the prior, the most recent, let's say 10 days or something like that. So that's my question on it. I'll delete my drawings there.   0:29:46.3 JD: Yeah, I think that's a fair question. So, the reason I didn't shift those before, despite you do see a pattern, so before the dotted line, I considered that period a baseline period where we were just collecting 'cause they hadn't tried anything yet. So Dr. Wheeler has these series of four questions. So in addition to seeing a signal, he's got these other sort of questions that he typically asks and that they're yes/no questions. And you want the answer to all those to be yes. And one of 'em is like, do you know why an improvement or a decline happened? And if you don't, then you really shouldn't shift the limits. So that's why I didn't shift them before. I chose not to shift them until we actually did something, actually tried something.   0:30:33.2 AS: Which is basically saying that you're trying to get the voice of the students, a clear voice, and that may be that over the time of the intervention, it could be that the... Sorry, over the time of the initial data gathering, that the repetition of it may have caused students to feel more joy in the classroom because they were being asked and maybe that started to adjust a little bit up and there's the baseline, so. Yep. Okay.   0:31:01.6 JD: Yeah. And so this is sort of where the project ended for the fellowship that Jessica was doing. But, what would happen if we could sort of see what happened, further out in the school year is that, either Jessica and the class could then be sort of satisfied with where the joy in learning is at this point where the improvement occurred. Or they could run another cycle, sort of testing, sort of a tweaked version of that noise reduction PDSA, that intervention or they could add something to it.   0:31:43.0 AS: Or they could have run another fishbone point, maybe the noise wasn't actually the students thought it would be the number one contributor, but, maybe by looking at the next one they could see, oh, hey, wait a minute, this may be a higher contributor or not.   0:32:01.2 JD: Yeah. And when you dug into the actual plan, the specifics of the plan, how that noise deduction was going to work, there may be something in that plan that didn't go as planned and that's where you would have to lean on, 'cause we've talked about the three sort of parts of the improvement team that you need. You need the frontline people. That's the students. You need the person with the authority to change the system. That's Jessica. And then someone with the knowledge of the system, profound knowledge. That's me. Well, those, the Jessica and her students are the one in that every day. So they're gonna have learning about how that intervention went, that would then inform the second cycle of the PDSA, whatever that was gonna be, whatever they're gonna work on next. The learning from the first cycle is gonna inform that sort of next cycle.   0:32:51.4 JD: So the idea is that you don't just run a PDSA once but you repeatedly test interventions or change ideas until you get that system where you want it to be.   0:33:01.1 AS: So for the listeners and viewers out there, I bet you're thinking gosh, Jessica's pretty lucky to have John help her to go through this. And I think about lots of things that I want to talk to you about [laughter] about my testing in my own business, and I know in my own teaching, but also in my business. So that I think is one of the exciting things about this is the idea that we just, we do a lot of these things in our head sometimes. I think this will make a difference and, but we're not doing this level of detail usually in the way that we're actually performing the tests and trying to see what the outcomes are.   0:33:43.9 JD: Yeah I think that for school people too, I think when we've attempted to improve schools, reform schools, what happens is we go really fast and the learning actually happens very slowly and we don't really appreciate what it actually takes to change something in practice. And what happens then is to the frontline people like teachers... The reformers have good intentions but the people on the front line just get worn out basically, and a lot of times nothing actually even improves. You just wear people out. You make these big changes go fast and wide in the system and you don't really know exactly what to do on the ground because the opposite is having Jessica's classroom. They're actually learning fast but trying very small changes and getting feedback right in the place where that feedback needs to be given right in the classroom and then they can then learn from that and make changes.   0:34:49.8 JD: And again, it may seem smaller. Maybe it doesn't seem that revolutionary to people but to me, I think it's a completely revolutionary, completely different way to do school improvement that actually kind of honors the expertise of the teacher in the classroom, it takes into account how students are experiencing a change and then I'm kind of providing a method that they can use to then make that classroom better for everybody so and I think in doing so students more likely to find joy in their work, joy in their learnings, teachers more likely to find joy in their work as well. So to me it's a win-win for all those involved.   0:35:34.9 AS: Fantastic. Well, should we wrap up there?   0:35:40.6 JD: Yeah, I think that's a good place to wrap up this particular series.   0:35:45.1 AS: And maybe you could just review for the whole series of what we've done just to kind of make sure that everybody's clear and if somebody just came in on this one they know a little bit of the flow of what they're gonna get in the prior ones.   0:36:00.4 JD: Yeah. So we did six episodes and in those six episodes we started off just talking about what do you need to have in place for healthy goal setting at an organizational level, and we put four conditions in place that before you ever set a goal you should have to understand the capability of your system, you have to understand the variation within your system, you have to understand if the system that you're studying is stable, and then you have to have a logical answer to the question by what method. By what method are you gonna bring about improvement or by what method you're gonna get to this goal that you wanna set. So we talked about that, you gotta have these four conditions in place and without those we said goal setting is often an act of desperation.   0:36:49.7 JD: And then from there what we did is start talking about these 10 key lessons for data analysis so as you get the data about the goal and you start to understand the conditions for that system of process we could use those 10 data lessons to then interpret the data that we're looking at or studying and then we basically did that over the first four episodes. In the last few episodes what we've done is look at those lessons applied to Jessica's improvement project and that's what we just wrapped up looking at those 10 lessons.   0:37:23.7 AS: I don't know about the listeners and viewers but for me this type of stuff just gets me excited about how we can improve the way we improve.   0:37:33.4 JD: Yeah. For sure.   0:37:34.9 AS: And that's exciting. So John, on behalf of everyone at the Deming Institute I want to thank you again for this discussion, and for listeners, remember to go to deming.org to continue your journey. You can find John's book Win-Win W. Edwards Deming, the System of Profound Knowledge and the Science of Improving Schools on amazon.com. This is your host Andrew Stotz, and I'll leave you with one of my favorite quotes from Dr. Deming, "People are entitled to joy in work."

In this episode, John Dues and Andrew Stotz apply lessons five through seven of the 10 Key Lessons for implementing Deming in classrooms. They continue using Jessica's fourth-grade science class as an example to illustrate the concepts in action.  TRANSCRIPT 0:00:02.2 Andrew Stotz: My name is Andrew Stotz and I'll be your host as we continue our journey into the teachings of Dr. W. Edwards Deming. Today I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. This is episode five about goal setting through a Deming lens. John, take it away.   0:00:23.2 John Dues: Yeah, it's good to be back, Andrew. Yeah, like you said, for the past few episodes we've been talking about organizational goal setting. We covered four healthy conditions, or four conditions of healthy goal setting and 10 key lessons for data analysis. And then what we turn to in the last episode is looking at an applied example of the 10 key lessons for data analysis and in action. And, if you remember from last time we were looking at this improvement project from Jessica Cutler, she's a fourth grade science teacher, and she did the improvement fellowship here at United Schools Network, where she learned the tools, the techniques, the philosophies, the processes behind the Deming theory, continual improvement, that type of thing. And in... And in Jessica's specific case, in her fourth grade science class, what she was settled on that she was gonna improve was, the joy in learning of her students. And we looked at lessons one through four through the eyes or through the lens of her project. And today we're gonna look at lessons five through seven. So basically the next, uh, the next three lessons of those 10 key lessons.   0:01:34.8 AS: I can't wait. Let's do it.   0:01:37.3 JD: Let's do it. So lesson number five was: show enough data in your baseline to illustrate the previous level of variation. Right. So the basic idea with this particular lesson is that, you know, let's say we're trying to improve something. We have a data point or maybe a couple data points. We wanna get to a point where we're starting to understand how this particular concept works. In this case, what we're looking at is joy in learning. And there's some different rules for how many data points you should, should have in a typical base baseline. But, you know, a pretty good rule of thumb is, you know, if you can get 12 to 15, that's... That's pretty solid. You can start working with fewer data points in real life. And even if you just have five or six values, that's gonna give you more understanding than just, you know, a single data point, which is often what we're... What we're working with.   0:02:35.6 AS: In, other words, even if you have less data, you can say that this gives some guidance.   0:02:40.9 JD: Yeah.   0:02:41.1 AS: And then you know that the reliability of that may be a little bit less, but it gives you a way... A place to start.   0:02:46.9 JD: A place to start. You're gonna learn more over time, but at least even five or six data points is more than what I typically seen in the typical, let's say, chart where it has last month and this month, right? So even five or six points is a lot more than that. You know, what's... What's typical? So I can kind of show you, I'll share my screen here and we'll take a look at, Jessica's initial run chart. You see that right?   0:03:19.3 AS: We can see it.   0:03:21.2 JD: Awesome.   0:03:22.3 AS: You wanna put it in slideshow? Can we see that? Yeah, there you go.   0:03:24.9 JD: Yeah, I'll do that.   0:03:25.4 AS: Perfect.   0:03:26.3 JD: That works better. So, you know, again, what we're trying to do is show enough data in the baseline to understand what happened prior to whenever we started this improvement effort. And I think I've shared this quote before, but I really love this one from Dr. Donald Berwick, he said "plotting measurements over time turns out in my view to be one of the most powerful things we have for systemic learning." So what... That's what this is all about really, is sort of taking that lesson to heart. So, so you can look at Jessica's run chart for "joy in science." So just to sort of orient you to the chart. We have dates along the bottom. So she started collecting this data on January 4th, and this is for about the first 10 days of data she has collected. So she's collected this data between January 4th and January 24th. So, you know, a few times a week she's giving a survey. You'll remember where she's actually asking your kids, how joyful was this science lesson?   0:04:24.4 JD: Mm-hmm.   0:04:27.2 JD: And so this is a run chart 'cause it's just the data with the median running through the middle, that green line there, the data is the blue lines connected by, or sorry, the blue dots connected by the points and the y axis there along the left is the joy in learning percentage. So out of a hundred percent, sort of what are kids saying? How are kids sort of evaluating each of these science lessons? So we've got 10 data points so far, which is a pretty good start. So it's starting to give Jessica and her science class a decent understanding about, you know, when we, you know, define joy in science and then we start to collect this data, we really don't have any idea what that's gonna look like in practice. But now that she started plotting this data over time, we have a much better sense of what the kids think of the science lessons basically. So on the very first day...   0:05:25.4 AS: And what is the... What is the median amount just for the listeners out there that don't see it? What would be the... Is that 78%?   0:05:33.8 JD: Yeah, about 78%. So that very first day was 77%. The second day was about 68%. And then you sort of see it bounce around that median over the course of that, those 10 days. So some of the points are below the median, some of the points are above the median.   0:05:50.4 AS: And the highest point above is about 83, it looks like roughly around that.   0:05:54.4 JD: Yeah. Around 82, 83%. And one technical point is at the point that it's a run chart we don't have the process limits, those red lines that we've been taking a look at and with a run chart and, you know, fewer data points, we only have 10. It's fairly typical to use the median, just so you know, you can kind of better control for any outlier data points which we really don't have any outliers in this particular case but that's just sort of a technical point. So, yeah, I mean, I think, you know what you start to see, you start to get a sense of what this data looks like, you know, and you're gonna keep collecting this data over an additional time period, right? And she hasn't at this point introduced any interventions or any changes. Right now they're just learning about this joy in learning system, really. Right.   0:06:51.8 JD: And so, you know, as she's thinking about this, this really brings us to... To lesson six, which is, you know, what's the goal of data analysis? And this is true in schools and it's true anywhere. We're not just gonna look at the past results, but we're also gonna, you know, probably more importantly, look to the future and hopefully sort of be able to predict what's gonna happen in the future. And, you know, whatever concept that we're looking at. And so as we continue to gather additional data, we can then turn that run chart from those initial 10 points into a process behavior chart. Right. You know, that's a, sort of a, you know, it's the run chart on steroids because not only can we see the variation, which you can see in the run chart, but now because we've added more data, we've added the upper and lower natural process limit, we can also start to characterize the type of variation that we see in that data.   0:08:00.1 AS: So for the listeners, listeners out there, John just switched to a new chart which is just an extension of the prior chart carrying it out for a few more weeks, it looks like, of daily data. And then he's added in a lower and upper natural process limit.   0:08:18.9 JD: Yeah. So we're still, we're still plotting the data for joy in science. So the data is still the blue dots connected by the blue lines now because we have 24 or so data points, the green line, the central line is the average of that data running through the data. And we have enough data to add the upper and lower natural process limit. And so right now we can start to determine do we only have natural variation, those everyday ups and downs, that common cause variation, or do we have some type of exceptional or special cause variation that's outside of what would be expected in this particular system. We can start making...   0:09:00.7 AS: Can you...   0:09:02.2 JD: Go ahead.   0:09:02.8 AS: I was gonna... I was gonna ask you if you can just explain how you calculated the upper and lower natural process limits just so people can understand. Is it max and min or is it standard deviation or what is that?   0:09:18.3 JD: Yeah, basically what's happening is that, so we've plotted the data and then we use that data, we calculate the average, and then we also calculate what the moving range, is what it's called. So we just look at each successive data point and the difference between those two points. And basically there's a formula that you use for the upper and lower natural process limits that takes all of those things into account. So it's not standard deviation, but it's instead using the moving, moving range between each successive data point.   0:09:52.9 AS: In other words, the data that's on this chart will always fall within the natural upper and lower. In other words it's... Or is, will data points fall outside of that?   0:10:05.7 JD: Well, it depends on what kind of system it is.   0:10:07.8 AS: Right. Okay.   0:10:09.8 JD: If it's a stable system, that means all we see is sort of natural ups and downs in the data. And we use those formulas for the process limits. The magnitude of the difference of each successive data point is such that it's not necessarily big or small, it's just based on what you're seeing empirically. It's basically predictable. Right. And if it's not predictable, then we'll see special causes. So we'll see special patterns in the data. So I think maybe last time we talked about the three patterns, or you know, in some episode we talked about the patterns that would suggest there's a special cause that goes to the study. Those three patterns that I use are, is there a single one of these joy in science data points outside of either the upper or lower natural process limit that'd be a special cause.   0:11:05.4 JD: If you see eight data points in a row, either above the central line or below the central line, that's a special cause. And if I see three out of four in a row that are either closer to the upper limit or to the lower limit than they are to that central line, that's a pattern of the data that suggests a special cause. So we don't, in this particular dataset, we don't see any special causes. So now we have... Now we have a very solid baseline set of data. We have 24 data points. And when you're using an average central line and get... Getting technical, once you get to about 17 data points, those upper and lower natural process limits start to solidify, meaning they're not gonna really change too much 'cause you have enough data unless something really significant happens. And then if you're using the median, that solidification happens when you get to about 24 data points.   0:12:07.5 JD: So when you're, you know, when you're getting to 17 to 24 data points in your baseline, you're really getting pretty solid upper and lower national process limits. So, as of this March 1st date, which is the last date in this particular chart, there are 24 data points. So you have a pretty solid baseline set of data. Right now, the upper natural process limit is 95%. That lower limit is sitting at 66%, and then the average running through the middle, that green line is 81%. So this basically tells us that if nothing changes within Jessica's fourth grade science system, her classroom, we can expect the data to bounce around this 81% average and stay within the bounds of the limit. So we would call this a common cause system because we don't see any of those rules that I just talked about for special causes. And that's important.   0:13:07.4 JD: So do we have an unstable system or a stable system? We have a stable system. A stable system means that the data is predictable and unless something happens, you know, and this could be something that happens in the control of the teacher in the class, or it could be out of the control of the teacher in the class, but unless something happens that's significant, this data is just kind of keep humming along like this over the course of March, April, May of this particular school year. Right. So once we get to this point, so we have baseline data we've collected in a run chart, we start to understand how that data is moving up and down. We got some more data and we added the upper and lower natural process limits. Now we can assess not only the variation, but also the stability and the capability of the system, all of those things, those questions can start to be answered now that we have this process behavior chart.   0:14:09.3 JD: And this brings us to the final lesson for today, which is lesson 7, which is the improvement approach depends on the stability of the system under study. So that's why one of the reasons why the process behavior chart is so powerful is because now I have an understanding of what I need to do, like what type of approach I need to take to improve this particular system. Right? So in this particular case, I have a predictable system. And so the only way to bring about meaningful improvement is to fundamentally change this science system, right?   0:14:52.6 JD: The flip side would be if I did see a special cause let's say, it was an unpredictable system. We saw special cause on the low side. I'd wanna study that, what happened on that particular day. Because if I see a special cause, let's say on February 2nd I saw a special cause, let's say I saw a single data point below the lower natural process limit that's so different and unexpected, I'd actually wanna go to her classroom and talk to her in her class and say, okay, what happened on that day? I'm gonna try to remove that special cause. Study of that specific data point is warranted. If you don't see those special causes, then those, even though there are ups and downs, there are increases and decreases. They're within that, you know, the expected bounds of this particular system. Right.   0:15:46.9 AS: And I was gonna say, I can't remember if I got this from Dr. Deming or where it came from, but I know as an analyst in the stock market analyzing tons and tons of data in my career, I always say if something looks like a special cause or looks strange it's probably an error.   [laughter]   0:16:03.2 AS: And it could just be for instance, that a student came in and they didn't understand how to fill it out or they refused to fill it out or they filled out the form with a really bizarre thing, or maybe they thought that number 10 was good and number one was bad, but in fact on the survey it was number one that was good and number 10 that was bad. And you find out that, you know, that special cause came from some sort of error.   0:16:26.6 JD: That's certainly possible. That's certainly possible.   0:16:29.5 AS: As opposed to another special cause could be, let's just say that the school had a blackout and all of a sudden the air conditioning went off for half of the class and everybody was just like really frustrated. They were burning hot. It was really a hot day and that special cause could have been a legitimate cause as opposed to let's say an error cause but you know, it causes an extreme, you know response on the survey.   0:16:56.9 JD: Yeah. And the thing is, is yeah, it could be a number of different things. Maybe she tried, maybe she had gotten some feedback about her lessons and maybe even she tried a different lesson design and it was new to her and it just didn't work very well. Maybe she tried to use some new technology or a new activity and it just didn't go well. But you know, if I'm seeing that data show up as a special cause and let's say I'm seeing that the next day or a couple days later, it's still fresh in my mind and I can even go into my chart and label what happened that day. Okay. And I... Now, okay, I'm gonna remove that thing or I'm, you know, if it's a lesson I'm trying, maybe I don't wanna give up on it, but I know I need to improve it 'cause it led to some issues in my classroom, but it's close enough to the time it actually happened that I actually remember what happened on that particular day and I can sort of pinpoint that issue.   0:17:52.9 AS: Yeah.   0:17:54.5 JD: And the data told me it was worth going into studying that particular data point because it was so different than what I had seen previously in this particular 4th grade science system.   0:18:06.5 AS: Makes sense.   0:18:09.9 JD: But in this case, we don't see that, that was a hypothetical. So all we see is sort of the data moving up and down around that green average line. So we have a stable system. So again, that tells me I need to improve the science system itself. There's no special causes to remove. So, the next question I think I would ask, and if you remember one of the data lessons is that we sort of combine the frontline workers, which is the students in this case. We have the manager or the leader, that's the teacher, and then someone with profound knowledge from the Deming lens, that's me, we're bringing these people together and we're saying, okay, you know, we're seeing this hum along this joy in science thing, hum along at sort of like an 81% average. So I think it's a reasonable question to ask, is that good enough? And should we turn our attention to something else. Now, there could be some situations where it's not good enough or some situations where that is good enough. They chose to keep moving to improve that joy in learning. But I think it'd be perfectly reasonable in some context to say, well, you know, sure, maybe we could get a little better here, but maybe it's not worth the effort in that particular area. Maybe we're gonna turn our attention to something else. You know.   0:19:23.7 AS: So you learn something from the chart and that could be...   0:19:26.4 JD: Learn something from the chart. Yeah, yeah.   0:19:27.9 AS: Because when I look at this chart, I just think hard work is ahead.   0:19:31.2 JD: Yeah. Yeah.   0:19:34.7 AS: 'Cause in order to, if you have a stable system with not a lot of extreme... Firefighting is kind of a fun thing, right? When you got special causes, you feel really important. You go out there, you try to figure out what those individual things are, you're the hero. You fix it, you understand it, you see it, whatever. But then when you get a stable system, it's like, oh man, now we got to think about how do we make some substantial changes in the system. It doesn't have to be substantial, but how do we make changes in the system, you know? And then measure whether that has an impact.   0:20:06.4 JD: Yeah. And to your point about fire... Fighting fires, like I didn't know, we had never measured joy in learning like this before, so I didn't know what we were gonna get with Jessica. And so you know what I think you also see here is a pretty well-run classroom. These are kids that are finding a pretty high amount of joy in their lessons. I think that you can kind of objectively say that, but they did choose to move on with the project and keep focusing on this particular system. And I thought it was really interesting. They actually... I'll flip slides here.   0:20:45.6 JD: They actually made this sort of rudimentary fishbone diagram, so you can, if you're viewing the video here you can see that Jessica just took a pen and a piece of paper and put this on the overhead in the classroom, and basically just drew a fishbone. And on the fishbone diagram is also called a cause and effect diagram. So out on the right it says effect. And she wrote low enjoyment, so she's meaning low enjoyment of science class. And they started brainstorming, those are the bones of the fish, what's leading to what's causing the effect of low enjoyment in science class. And so they... She did this brainstorming activity with the kids. So some of the things they came up with were why is there low enjoyment with science class? Well, the computers are sometimes lagging when the kids are trying to use them. They're mad at Ms. Cutler for one reason or another. There's a lot of writing in a particular lesson. There's a lot of reading in a particular lesson.   0:21:58.2 AS: Other teachers coming into the room.   0:22:00.7 JD: Other teachers coming into the room and disrupting the class.   0:22:02.7 AS: Stop bothering me.   0:22:04.1 JD: Yeah. I mean, you know, these are the things you don't often think about. And then they talked about classmates making noises throughout classes, another distraction. And they basically categorized these into different categories. So there were sort of things that made the lesson boring. That was one category. Accidents happening, those are like the computers not working correctly. Scholar... We call our student scholars. So students getting in trouble was one, and then distractions was another category. And so then they did another activity basically after they had this fishbone. And they basically did like a voting activity where they would figure out which of these is the most dominant cause of low enjoyment. And actually what they came up with is their classmates call, like making noises, like students making a lot of noise, making noises, random noises throughout the lesson, they identified that particular thing as the thing that they're gonna then do something like design a plan, do study around, like how are we gonna reduce the amount of noise in the class?   0:23:12.7 JD: And this is all the students coming up with these ideas. Of course, Jessica's guiding these conversations as the adult in the room, but the kids are coming up with this. Like I never would have, well, maybe I shouldn't say I would never have, but it probably wouldn't likely have been on my radar that teacher, other teachers coming into the room was a main source of distraction. You know, who knows what they're doing, dropping off papers that have to be passed out, that dismissal or coming to find a kid for this thing or that thing. Who knows why they're stopping by. But schools are certainly rife with all kinds of disruptions, announcements, people coming into the room, those types of things.   0:23:51.0 AS: It's interesting too to see mad at Miss Cutler because... I was just reading a book about or some research about how to get rid of anger and that type of thing. And they talk about meditation and I do breathing exercise before every class, when every class starts. And it's a way of just kind of calming down and separating the class time from the chaos of outside, but it also could be something that could help with feeling mad.   0:24:27.9 JD: Yeah. And I think if in certain classrooms that certainly could have risen to the top. And then what you do is then design the PSA around that. So how do you do meditation? How do you know if you're going to do... How do you know if you're doing it right? How long do you do it? You know? Does it have the intended impact? You could study all kinds of different things with meditation, but...   0:24:52.4 AS: And are you really mad or is there... Are you really mad at Ms. Cutler or are you... Are you frustrated about something else? Or that...   0:24:58.1 JD: Exactly. Yeah. Is it warranted? Is there actually something that she should stop doing or start doing? There's all kinds of possibilities there. But the main point, and I think this kind of would bring us to the wrap up is taking this approach is very different. Even just the step, Jessica's step of saying, I'm gonna work on joy in science, joy in learning in science class. That's a very different approach. And then step beyond that, I'm gonna involve my students in this improvement approach. And we have these various methods and tools for systematically collecting the classes input, and that those are improvement science or continual improvement tools that we're using. And then we're applying some of the knowledge about variation, Deming sort of data methods to understand that data, that we've systematically collected from students.   0:25:58.7 JD: And now students are involved. So they're actively coming up with both the reasons, the problems that are happening. And then they're like what we'll get into in the last few lessons is their input into the solutions, the change ideas that are gonna make things better. But all of this represents a very different approach than what's typical when it comes to school improvement. These things are not being handed down from on high from someone that has no connection to this classroom whatsoever. Instead, it's actually the people in the classroom that are developing the solutions.   0:26:36.9 AS: I just was thinking about the idea of imagining that this group of students is working really hard on that, and they come up with so much knowledge and learning about how to create a more joyful classroom. And then imagine that they've now codified that together with Ms. Cutler to create kind of the standard operating procedures. Like we put up a sign on the door outside that says, do not disturb until class is over, or...   0:27:06.1 JD: Something simple. Yeah.   0:27:07.0 AS: And that they come up with, and a breathing exercise or whatever that is. And then you imagine the next group of students coming in for the next year, let's say, or whatever, that next group who can then take the learning that the first group had and then try to take it to another level, and then upgrade how the operations of the room is done. And you do that a couple of iterations, and you've now accumulated knowledge that you are building on until in a business, you're... You're creating a competitive advantage.   0:27:40.4 JD: Yeah, absolutely. And another thing that these guys did was they didn't say we're gonna improve X, Y, or Z and then set an arbitrary goal, which is one of the things we've talked about that often happens at the outset of any type of improvement. They didn't... They sort of avoided this act of desperation. We talk about goal setting as an active... Are often goal setting is often an act of desperation. They avoided that completely. Instead, what they did was we gathered some baseline data to understand what is the capability of our system when it comes to joy in learning. That's what they did first. They didn't set the goal first. A lot of wisdom, a lot of wisdom in 10 year olds for sure.   0:28:22.1 AS: That's interesting. Well, John, on behalf of everyone at the Deming Institute, I want to thank you again for the discussion and for listeners, remember to go to deming.org to continue your journey. You can find John's book Win-Win: W. Edwards Deming, the System of Profound Knowledge and the Science of Improving Schools on amazon.com. This is your host, Andrew Stotz, and I'll leave you with one of my favorite quotes from Dr. Deming, and its absolutely applicable to today's discussion. People are entitled to joy in work.

Bill Bellows, who has spent 30 years applying Dr. Deming's ideas, discusses testing for understanding transformation with Andrew. They talk about little tests to measure learning impact and share amusing anecdotes. Topics include effective questioning, incentives in motivation, and fostering comprehension during transformation processes.

Bill Bellows shares his experience of implementing Dr. Deming's ideas with a small group at a large company. They discuss transparency, decision-making, and the importance of open communication to drive organizational change.

Can a 4th grade class decide on an operational definition of "joy in learning"? In part 4 of this series, educator John Dues and host Andrew Stotz discuss a real-world example of applying Deming in a classroom. This episode covers the first part of the story, with more to come in future episodes! 0:00:02.3 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we continue our journey into the teachings of Dr. W. Edwards Deming. Today, I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. This is episode four about goal setting through a Deming lens. John, take it away.   0:00:22.6 John Dues: Good to be back, Andrew. Yeah, we've been talking about organizational goal setting last few episodes. A couple episodes ago, we talked about those four conditions that organizations should understand prior to setting a goal. Then we sort of introduced this idea of trying to stay away from arbitrary and capricious education goals. And then we got into these 10 lessons for data analysis. And so what I thought we could do now is we've got that foundation in place is that we could take a look at an applied example in real classrooms of those 10 key lessons in action to kind of bring those alive. And I ran this project a few years ago with a teacher named Jessica Cutler. She's a fourth grade science teacher in our network. And she was going through something we call a Continual Improvement Fellowship. So we do this sort of internal fellowship where people can learn that sort of way of thinking, the tools, techniques, the theories related to the science of improvement. And then they actually take that right away and apply it to a problem in their classroom or their department or their school, depending on who it is.   0:01:55.0 JD: And so what Jessica was doing, what her project ended up being was she was trying to improve the joy in learning in her fourth grade science class. So it's interesting to see how that sort of project evolved. So I thought we could revisit each of the 10 lessons and how that lesson was applied in Jessica's improvement project. And we'll maybe get through three or four of the lessons today. And then over the course of the next few episodes, kind of get to all 10 lessons and think through how they were... How that went in her improvement project.   0:02:08.1 AS: Sounds like a good plan, practical application.   0:02:12.0 JD: Yeah. I mean, it was interesting too, because she didn't initially sort of consider joy as a possibility. She was thinking like, I'm gonna work on improving test scores or something like that was sort of her initial brainstorm. And then sort of pivoted to this when we kind of talked through what was possible from the Deming philosophy type of standpoint. So it's interesting to see how things evolve. But just to kind of revisit, so we talked through these 10 lessons. Lesson one was "data has no meaning apart from their context." So we talked about these questions that are important, like who collected the data? How was it collected? When was it collected? Where was the data collected? What are the values themselves represent? What's that operational definition for the concept under measurement? Have there been any changes to that operational definition as the project unfolds? And so even with a project with a teacher and her students, all of those questions are relevant. They're still important just because you're dealing with students that doesn't mean anything changes on that front. So it was important for her to sort of think through all of those things as she thought through the start of her project.   0:03:28.9 JD: And what her and her students came up with after they sort of decided that they were gonna focus on joy, they focused on this problem statement. And they were like, well, what do we want science class to look like? 'Cause that was sort of their starting point. And what her and her students...Oh sorry go ahead.   0:03:45.9 AS: One thing you started off talking about her, now you're talking about her students. So she got her students involved in this process. Is that what you're saying?   0:03:56.2 JD: Yeah. So they were working together from the very outset even...   0:04:02.0 AS: As opposed to a teacher talking through this with a principal or something in a faculty room and then thinking of how do I... Okay.   0:04:09.2 JD: Yep. That's right. Yeah. And so what they came up with is the sort of desired future state of science classes. "We are able to stay focused through science, enjoy science class and remain engaged." And so to give some context, what was happening is that she taught science and social studies and it was sort of like a back-to-back class period. And they would do science second. And so by the time they were doing science, sometimes the students were getting off task, disengaged. They weren't as engaged as either the students wanted to be or the teacher wanted them to be in that second lesson. So they, they came up with that as the thing they were gonna focus on. And then because they were gonna focus on joy in learning, they had to define what that meant. So what did joy in learning mean to that fourth grade science class? And what they came up with as a definition, which I really like, is "we wanna have fun learning, finding things we like to learn and have fun completing classwork and activities." So they came up with this operational definition. And keep in mind, these are fourth graders and Jessica's having these conversations like, what's the operational definition? That's not probably typical language you're gonna use with fourth graders. But if you walk them through these things, they actually pick up on it pretty quickly.   0:05:26.1 JD: It's actually pretty cool to see.   0:05:28.1 AS: And to them, a more simpler word sounds like was fun.   0:05:35.0 JD: Yeah, right. They wanted that to be a part of the science learning process. So basically, once they had the operational definition, they had to think through, well, how are we going to measure that concept that we've defined? And what they did was they just developed a simple survey. Jessica did it in Google Forms. just had, really just had two questions. The first question was, on a scale of 1 to 10, how much did you enjoy science class today? And then there was a second open-ended question that said, what made you enjoy or not enjoy class today? So it was fresh in the kids' minds. So basically, at the end, each kid has a Chromebook in Jessica's science class. She would just sort of share the link to the survey, and the kids would complete that as the closing activity for the lesson. So she would get two things out of it. So 1 to 10, just a real quick sort of numerical quantified value, how much the kids enjoyed science class that day. And then, because it had just happened, the students could say what they did and didn't like about the lesson. Oh, we haven't used computers in a few days. Or it'd be nice if I had a video to help bring this concept alive. Or there's a few words that you use that I don't know the definitions to. Could you add those definitions to the glossary? So just things like that, simple things like that.   0:06:55.9 JD: Right away. And then what Jessica could then do is take that information and actually adjust her lessons as she planned maybe for the next week, she could make those adjustments based on this feedback she was getting from the students. So that's sort of the application of lesson number one. So what are we measuring? How are we gonna measure it? When are we collecting this data? That type of thing. Lesson two, if you remember back from when we covered the lessons was "we don't manage or control the data. The data is the voice of the process," right?   0:07:28.9 JD: So we talked about this ideas that while we don't control the data, we do manage the system and the processes from which the data come, right? So, and this is really key conception of the system's view. You, you say you're going to improve this particular classroom. So that's the system. So you're not necessarily controlling the data. You're not controlling how the kids are evaluating, the numbers that they're putting one through 10 to assess joy in learning, but what the teacher and then the students, because of this project do have control over are the learning processes that are happening throughout science class, right? And so back to your point about you switch from talking about Jessica, the teacher to the students. And then you said "we" that's also a key conception of taking this approach, right?   0:08:24.4 JD: So what I think Deming would say is that when you're going to improve an organization, you have to sort of combine sort of three critical pieces. One thing is you need someone from the outside, from outside the system that has Profound Knowledge. And then that person or persons has to be collaborating with the people working in the system. So those are the students, they're working in the science class system. And, then you that third group or that third person is the manager or managers have that have the authority to work on the system. So in this case, Jessica has the authority to change what's happening in her science class.   0:09:10.2 JD: The students are the workers working in that science system. And then that third part is that person that has the sort of understanding of the System of Profound Knowledge and it's sort of bringing all of these parts together that really is how you begin to transition sort of conventional classrooms to those guided by the Deming quality learning principles, right?   0:09:33.1 JD: So in in the case of Jessica's project, that person that was, that had a System of Profound Knowledge lens was me. So I was sort of acting as an, the outsider, 'cause I'm outside of the science system. But I have this understanding of the System of Profound Knowledge. And I'm working with Jessica as she's working with her students, to sort of bring that lens to the projects.   0:10:00.4 AS: And what's the point of doing all that if she doesn't have the ability to make the changes necessary to test, if you're gonna if we change this, it's gonna result in something why go and do all this if you're just stuck in a system that you simply cannot change because of government regulation or whatever, maybe.   0:10:17.5 JD: Right. Yeah. So it's bringing all those pieces together. But what I found thinking about the three parts of a team that's working toward organizational improvement, what I've found in the past is, in my experience, whether it's a school improvement team or a district based improvement team, most of them are devoid of at least one parts of one of those components, usually two of those components, 'cause usually students aren't involved.   0:10:45.9 JD: And then in most school systems, there's no one with this outside knowledge, the System of Profound Knowledge lens, right. And I think it's what we're really doing is the students can identify the waste, the inefficiency, the things that aren't going well from their perspective, but we don't often ask them. Or if we do, we do it in a way where it's an end of year survey or an end of semester survey, but this is collecting that feedback in real time and then acting on it. We're not planning to do something next year with this feedback, we're actually planning to do something the next day, or maybe the next week, to adjust the science lessons.   0:11:24.8 AS: And it's one of those two things that come into my mind, what, how do you handle the idea that what's causing the impact on joy in learning could be that the student had a bad night, the night before. And I guess by doing many samples that starts to kind of wash out. And then the other question is since the students know that the teacher could likely make an adjustment, is there any possibility that they could be gaming or playing the system.   0:11:57.1 JD: Well, that's interesting, because I think, well on the first point. I think pretty quickly, my experience with this and David Langford I know you've talked to has echoed this sentiment is you know, he was working with high school students, this is an elementary project but either way. You may get some students that don't take this seriously. At first. And you may get some kind of crazy answers crazy brainstorms or crazy survey submissions, although I don't think Jessica got much of that.   0:12:30.2 JD: But in other projects I've gotten some stuff at the outset that was a little bit off the wall. But like David said to me when I first started this and then it's been my experience since is that kids, once they realize that you're actually gonna act on the feedback, as long as the feedback is in good faith. They actually start to take it seriously, pretty, pretty quickly. And so I think pretty quickly, those sort of types of worries go by the wayside. Now, I will say I did say that the...   0:13:01.1 JD: One of the components that has to be on this improvement team is the person that has the authority to change the system. So at the end of the day, even though we're gathering this input, Jessica's really the person as the teacher of that classroom that has the authority to make the changes to the system based on her judgment or, her professional judgment as a science teacher of what should happen. And so the students certainly offer feedback and inform that process, but ultimately it's Jessica that's gonna determine the changes to the system.   0:13:34.0 AS: I hear David in my ears saying, you know what, Andrew? You don't trust the students? They probably have a more honest, view of what's going on than most adults do. So yes, I hear the voice of David Langford.   0:13:49.2 JD: Yeah. Well, and interestingly, and we'll get into this towards the end, not today, but when we get to some of the other lessons, interestingly, not to give away the story, but, one of the things that was getting kids off track was a lot of noise during class, kids making noises. And they actually came up with this system where they were kind of penalizing each other. This was their own idea. And so, kids know exactly what's going on in class. And so it was interesting to see how they came up with some ideas to rectify that. But yeah, so it was really just bringing together, these three groups or, the group of students and then Jessica and then myself. It's that combination that's really where the power for improvement lies. And again, I, that type of partnership is just not typical in school improvement situations.   0:14:45.7 JD: So that's lesson two, applied. Lesson number three is "plot the dots for any data that incurs in time order." Right? So we've talked about this a lot. The idea behind the primary point of "plot the dots" is that plotting data over time helps us understand variation, and that in turn leads us to take more appropriate action. I think that what we decided to do with Jessica's project is, start plotting the points on a run chart and connect those points with a line, and then it becomes pretty intuitive as we're looking at that data, what joy in learning looks like in this science class. And then once we have enough data, we can turn that run part chart into a process behavior chart and actually add the limits.   0:15:40.2 JD: So, like I said, Jessica, once her and the class determined that what they were going to improve was join in learning, and they defined that concept operationally and created the survey, right away they started gathering this survey data as a part of the project, and usually they would gather the data maybe, two or three times a week across the course of this particular improvement project. So maybe I'll share my screen just so you can see what that initial run chart looked like. So, you have this run chart, and I left this in the spreadsheet so you could see the actual data. So as she began administering these surveys, she would send me the data and then I would create it the run chart for her, start plotting that data so that both of us could sort of see the variation in that survey data over time. And then she could actually take this, she would put this run chart on a slide, and every week or so she would actually show the students what the data looked like.   0:16:49.7 AS: And just to be clear, we've got a chart for those that are listening, we've got a chart that has a blue line and it's going up and down kind of around the level of about 79. So they've got points that are based, that are days. Some days are below that 79 some days are above. But also I'm assuming that those points are the output of all the surveys. So the average answer on that day from the survey as different from the average or median of all the day's output, correct?   0:17:31.1 JD: Yeah, that's right. So this is, the run chart from Jessica's class that's displaying the survey results. And what they're measuring is joy in science class as assessed by the students.   0:17:44.3 AS: On the first day, the students basically said, 75% of the respondents said that they had joy in science.   0:17:51.9 JD: That's right. So in this particular school year, which was two years ago, so we had done some of the project planning before kids went on winter break, and then when they came back from winter break, they were ready to start administering the survey. And we started plotting the dots, charting the data over time. So the X axis for those who are listening are the dates.   0:18:16.2 JD: The, Y axis is the joy in learning, percent of kids that the rating of the kids from one to 10. And then I just turn it into a percent. And so you have the green line, the central line running through is the median. We're using the median 'cause that's fairly typical for a run chart because typically run charts don't have as much data as a process behavior chart. And so, outliers can have a greater impact. So we're using the median to sort of control for that. Although this data's fairly tight. So on day one, like you said on January 4th of this school year 75, the kids sort of rated the joy in learning of that particular lesson as a 75% of 100. And then you sort of see it bounce around.   0:19:04.7 JD: That median of 79. And so what I'm showing is the data from the first 10 surveys that Jessica administered at the end of class. So over the course of 20 days from January 4th through January 24th, she administered that this survey 10 different times. So about two to three times a week. And so we see a high of about 83% joy in learning and a low of 67% joy in learning. And you have about half the points above the median, about half the points below the median. So even though it's only 10 data points, Jessica and her class, and then myself, we were starting to learn about what did joy in learning, joy in science class actually look like? Now that we have this definition and we're measuring it with these surveys and then plotting these data points. So again, she's actually putting this up on, on the, up on the screen so kids can actually see this. And what she said was after the 5th or 6th survey, and she's plotted this and put this up on a screen a few times, the kids are actually getting excited. And they're wanting to see their data. They're wanting to see what the results look like for each survey as she started plotting this.   0:20:30.6 AS: It's funny because I, when I was a loading supervisor at Pepsi, I started putting up the percent correct for each of the loaders in the warehouse. And I didn't make any comment or anything, I just put it up there. And yeah, people are interested when they start seeing numbers, they start thinking, they start asking questions.   0:20:54.3 JD: Yeah, and you can see too at a school, in a fourth grade science classroom, you can see all types of lessons, you can sort of build up this reading graphs, calculating percentages, using when do you use line graphs for some other type of graph?   0:21:10.2 AS: And why use median versus mean? Because a small amount of data could be distorted if you have a huge outlier.   0:21:18.8 JD: Yep, all kinds of practical lessons. So this brings us to sort of the last lesson for this particular episode. I think lesson four is two or three data points are not a trend, right? So, we've said that you should start plotting the dots as soon as you've decided to collect some type of data that occurs over time. And really when people ask me what type of data can you put on a run chart or a process behavior chart, there's almost any data that you're interested in improving in schools unfolds over time, almost all of it. Whether that's a daily cadence, a weekly cadence, monthly, quarterly, yearly, whatever it is, right? But the problem is the vast majority of data that we look at as educators and really probably most people, it's typically two or maybe three data points. But that doesn't tell you anything about how the data is varying naturally. So when we start thinking about this particular data, we start learning quite a bit. For one, as a teacher, I would have no idea how my kids would evaluate their joy in my classroom.   0:22:29.9 JD: And so I think if I was Jessica, I'd be pretty happy off of that, that the sort of average or the median rating is close to 80%, basically the rating each lesson has an eight out of 10. Right. I think a second thing is let's say we were a school district and we did systematically give our kids some type of survey that assess their satisfaction with the school. Right. Maybe they do it twice a year or annually. Right. And so after at the end of the year, you have two data points, but you don't really have any idea for what to do with that data. You have no idea if you collected three or four or five data points, what that would look like. And here she is in just 20 calendar days and a couple of school weeks. She's got 10 data points to work with already. So she's building that baseline of data. So I think what this is to me is just a very different approach to school improvement.   0:23:39.4 JD: And the tools are relatively simple. The ideas are relatively simple. But I think overall this really, the takeaway I want for folks is that this project really illustrates a very different approach to school improvement, guided by these sound sort of Deming principles for how to use data, to how to understand variation, to how to include the people working in the system, right?. We've talked about these arbitrary targets throughout this series, and you could see that when Jessica and her class would go to maybe set a goal for joy after collecting some of this data, that goal would be tied to something real. It's tied to actual data from the classroom. And you can sort of avoid goal setting as an act of desperation when you take this type of approach.   0:24:38.4 AS: Joy in the joy of bringing joy in science.   0:24:44.2 JD: Yeah, it's really all about this process, right? It's the kids getting into this process, that's the psychological part. They're involved in their educational process. And so that is completely different than what's happening in the typical classroom, I think, in the United States.   0:25:00.5 AS: You can imagine somebody not wanting to do this because they're afraid of what they're going to see.   0:25:07.0 JD: Certainly [laughter]   0:25:08.3 AS: Yeah.   0:25:08.7 JD: Certainly. Yeah. Hopefully they would be open to sort of collecting the data and being reflective as a professional. But I could see, maybe that's not, tha's not always the case. And another question, I kind of shared this project with some folks, in different settings, and one of the questions I typically get is, well, what about the science test scores? Like, this is great if kids have joy, I guess is kind of the reaction. But what... How does that impact the academics?   0:25:44.0 JD: And my response is, well if kids don't find joy in their learning and they're not engaged, what kind of results are you gonna get? [laughter] To me this is sort of like a part of the process that leads to academic outcomes, when you enjoy the things that you're doing, when you feel like you have some control over a process, maybe not the full control, but when you have some control, when you have input into something that you're doing all day long, you're gonna have more investment because, you know, because you're seeing that your input has meaning. That's really that psychological component.   0:26:18.2 AS: It's obvious, but maybe not proven.   0:26:21.9 JD: Yeah, I think so. I think so.   0:26:28.4 AS: Okay.   [SILENCE]   0:26:30.9 JD: Yeah. I think that's a pretty good spot to wrap up this opener with the... We covered those first four lessons and started to look at how this project unfolded in, Jessica's classroom. And I think, next we can kind of see as she gathered more data, what this looked like over time. And then as she sort of had that baseline in place, then the next thing we'll look at is: what did she do as a change idea or an intervention to try to make these rates go higher in her classroom?   0:27:08.4 AS: That's interesting. I mean, in my wrap up of this, I think how lucky, is Jessica to have someone from the outside? I think a lot of teachers and a lot of people in business, they don't really have anybody to go to. And the company's not providing that type of stuff or the school is not providing that. And so you just kind of make it up as you go along. And I think that's, that's one of the things, 'cause I'm, I think like probably other listeners and viewers who are, listening to this, they're thinking, I wonder if John could help me do that in my area? The idea of, we all know there's places that we could improve that we may not be. And if a school system can provide that, wow, that's a big... That's exciting.   0:27:56.6 JD: Yeah. I'd be happy to. And it was like a, it was definitely a mutual effort. Jessica put a lot of work into sort of, 'cause she has gone through that fellowship, she had to sort of learn all of these tools and then actually, turn around and put them into practice in her classroom. And she found ways to do this in a way where, she could still do the things she was required to do, like delivering the lessons that she was required to deliver and those types of things. But then she found ways to sort of incorporate what she learned in the fellowship to make her classroom better. Seeing that, seeing her openness to feedback that really made this like a, I think a, mutually beneficial experience. And I think the kids enjoyed it too.   0:28:39.2 AS: And the purpose of this series too is, the idea of how can you do this at home and how can you start doing it in your own school, in your own classroom, in your own life? And so I think I'm looking forward to the next session where we're gonna go deeper into... I've already got, a series of questions and things that I'm wondering, and then I saw some tabs in your, in your worksheet that I thought, okay, there's gonna be some more interesting stuff. So I think we're all gonna see you in that next section.   0:29:09.9 AS: And on behalf of everyone at the Deming Institute, I want to thank you again for this discussion and, taking the time to go through these steps with us. And for listeners, remember to go to deming.org to continue your journey. You can find John's book Win-Win, W Edwards Deming, the System of Profound Knowledge and the Science of Improving Schools on amazon.com. This is your host, Andrew Stotz. And I'll leave you with one of my favorite quotes from Dr. Deming, and it's particularly apropos, people are entitled to joy in work.