Plan-Do-Study-Act: Path for Improvement (Part 7)

Can you use Plan-Do-Study-Act (PDSA) during the information-gathering phase of an improvement project? Yes! Join John Dues and host Andrew Stotz as they discuss how John's team used PDSA to learn more about chronic absenteeism, their surprising findings, and what they'll do next.

TRANSCRIPT

0:00:02.8 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we dive deeper into the teachings of Dr. W. Edwards Deming. Today, I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. The topic for today is powerful learning with the PDSA cycle. John, take it away.

 

0:00:25.5 John Dues: Yeah, Andrew. It's good to be back. For the past two episodes or so, we've been working towards defining the problem of our chronic absenteeism issue, of course, we have a problem with chronic absenteeism, but we're trying to narrow that down and get a more specific problem statement. Last time we talked about how our improvement team, basically, had come to the conclusion after a few weeks of study that we didn't have enough information to write that specific, precise problem statement. So what we decided to do, and we started looking at this last time, was we started to gather additional information through a Plan-Do-Study-Act cycle. So that's what we'll focus on today, is this first PDSA cycle, and I think it's good to know that you can use PDSAs to run an experiment to test a new idea, but you can also run a PDSA to gather more information. Those are both very worthwhile uses of the PDSA cycle. So I go to share my screen just so I have that model up so that people who can see it, for those that are viewing. Can you see that now?

 

0:01:33.8 Andrew Stotz: Coming up. Okay, we can see it.

 

0:01:36.6 John Dues: All right, great. So you remember, we've been working through this four-step process for those who are hopping in for the first time or as a review for those that have been following along. So we have these four steps: set the challenge or direction, grasp the current condition, establish your next target condition, and then an experiment to overcome obstacles. And remember, we've been working through this team, that's a combination of people working in the system, people who have the authority to actually change the system, and then the System of Profound Knowledge coach. So I think that's a pretty powerful combination of people, and that's our team here working on this chronic absenteeism problem. You also remember that we have this long-range goal that this challenge that is to improve our chronic absenteeism from right around 50% to down closer to 5%, and I don't know if you remember this, but a number of episodes ago, I showed you the data we had over time, and we just had three years of data. Since that time I showed you that first run chart, I've actually gone back and added chronic absenteeism rates for our schools going all the way back to the 2016 - '17 year, and I think it's worth it to just take another quick look at those rates over time in a process behavior chart.

 

0:03:00.2 Andrew Stotz: Exciting.

 

0:03:01.5 John Dues: So, yeah. This is our chart. So we add more days so why not display it in this way. So what this chart is, is again a process behavior chart, we have school years going back to the 2016-'17 school year, and then through last school year. And we have the blue dots displaying the chronic absenteeism rate for each of those school years across our school system, and then the green is... The green line is that central line, it's the average of all years, the red lines are those natural process limits that sort of tell us where we can expect our data to fall given that this is a predictable system. So you can see right off the bat, something that's pretty obvious is that the first four years of data are below that central line, and then the last four years of data are above that central line. And of course, it's not too hard to sort of recognize that the pandemic happened towards the end of the 2019-'20 school year, and then sort of... We were all remote heading into that 2021 school year, and then for a number of years after we were in remote or hybrid, and so you can see very clearly that while there was chronic absenteeism in our system prior to the pandemic, after the pandemic, it exploded and it has not subsided.

 

0:04:28.7 John Dues: So in a typical year prior to the pandemic, we were somewhere around that 25, about a quarter of the kids give or take, depending on the year, of the kids were chronically absent, and then after the pandemic, we can see it sort of... Or at the begining of the pandemic, explodes up and then has settled around this, right about 50% average.

 

0:04:51.1 Andrew Stotz: And the fact that it's remained at this much higher level of, let's say 50-55% tells you that there's like... It has had somewhat of a permanent impact, whereas some people may think that the COVID situation caused a spike in chronic absenteeism up to 70% or whatever that number was, and then it came back to normal. But it's far away from normal.

 

0:05:26.4 John Dues: Yeah, and I haven't done a deep analysis. But in addition to the chronic absenteeism, the National Assessment of Educational Progress, which is like the gold standard, the report card for the nation, a nationally known test given every couple of years, that data shows that the 4th and 8th graders that take that test across the country in ELA and math, the scores are down coming out of the pandemic as well and have not rebounded. So I think that data is important. I'm not necessarily saying one way or the other, what we should have done, but what I am saying is like when we make decisions like shutting down schools, it's not just a decision that has an impact in the moment, there are ramifications on an ongoing basis. And we should sort of take that calculus into consideration when we're deciding what to do in a situation like that.

 

0:06:20.0 Andrew Stotz: And this also shows that you're taking on a pretty serious challenge because...

 

0:06:23.8 John Dues: Very serious. Yeah.

 

0:06:25.0 Andrew Stotz: Yeah, it's serious for the students, but it's also serious in the sense that it's been lingering at this very high level of chronic absenteeism, so, okay.

 

0:06:37.0 John Dues: Yeah.

 

0:06:37.5 Andrew Stotz: Shocking.

 

0:06:38.1 John Dues: Yeah, it is pretty shocking.

 

0:06:38.7 Andrew Stotz: That's not happening in Asia.

 

0:06:40.6 John Dues: No, and it's... I think a number of places in the United States, the learning chronic absenteeism has bounced back, but in the places where you expect where there's, especially high concentrations of poverty and things like that. It's sort of remained a serious issue even depending how you mark the end of the pandemic, two or three years after the primary part of the pandemic anyway. The height of the pandemic, if you will. So, ongoing challenges for sure. So what I said was that the team was going to run this initial PDSA cycle to gather more information, of course, there were some initial thoughts on why kids were missing so much school. We've talked about these transportation, different expectations that have been set for when to stay home, family and instability, those types of things. But again, we want to further test those assumptions early on in the project. So the key question that we were looking at is, for this first PDSA cycle, at least was will the combination of a what we call an empathy interview, which is just like where we sit down with a student or the family and try to better understand what's going on, and then daily attendance tracking was the other part of this, will that lead to a modest increase in the students average daily attendance rate during the period of the intervention.

 

0:08:11.9 John Dues: So even though we weren't necessarily testing a change idea, there was this sort of like... We framed it as a modest intervention in terms of sitting down with the kids and then doing this daily tracking and showing them the data. And a key part of this plan phase is we had all of our team members predict what they thought would happen with the four students that we chose to have those interviews with and track the daily attendance of during Cycle 1. So we had everybody really think through, "Okay, what do we think will happen when we put this plan in place?" And that's going to be really important because when we actually run the test, we want to compare the predictions to what actually happens, and that's where a lot of the learning happens from a PDSA cycle.

 

0:09:02.9 Andrew Stotz: And just for the listeners or viewers out there, why is it important to do that? Some people would say just do it and find out what the result is.

 

0:09:12.3 John Dues: Well, if you don't take a stance basically before the intervention happens or before the plan is put in place, then there's no learning that can really happen because whatever happens happens. But you didn't sort of say, "Here's what I think's going happen." And a lot of times, we quantify that prediction, and then what you can see is the difference between those two things is not only the learning, but it's also an indication of how well you understand your system. So what I mean is, if we put an intervention in place and I say, "Okay, I think this is going to have a 15% increased impact on whatever it is, a test score or attendance in this case," and then it has no impact, then I don't have an understanding really of what's going to work to fix whatever I'm trying to fix. But if the prediction bears out and it's pretty close to what actually happens, then that means, oh, I have a pretty good grasp on what's going on in my system. Yeah, kind of makes you put a stake in the ground, and it makes you mentally when you're doing it, it makes you think.

 

0:10:19.0 John Dues: Look further ahead and say, "Okay, if I do this, do I actually think this is going to be effective?" And you can also see the team's thinking. Some people might think this is going to be very effective, and some people may think it's not. Some people might think it'll work with some students, but not other students, and it gives you that picture black and white before you actually run the test.

 

0:10:37.8 Andrew Stotz: And in academic studies, it's really important to identify your end point that you're testing for. Otherwise, you run the risk of switching your end point as you get through your research because you're grounded initially.

 

0:10:53.2 John Dues: Right, exactly. Right, yep, absolutely. So in this plan, what we did was, this... We had parent conferences coming up, so we just said, that's a natural time to sit down with these four students that we chose at parent conferences. So at the end of November, we did that, we reviewed the data, we sat with the family to discuss some of the causes of the attendance challenges. We explained the plan to track attendance for 15 days coming back from Thanksgiving break. And then part of the plan was collecting that quantitative data, like the actual attendance rates each day for each kid, but then it was qualitative too, because we were asking the family on the front end, what was the sort of overall cause of the problem. And then we were asking the actual student every day like, "Oh, on this particular day, why were you absent, or why were you late, why did you miss school?" And we were tracking that across 15 days, so it's a relatively short time period, 'cause we don't want to go off on some grand experiment and then find out three months from now that our intervention wasn't effective.

 

0:12:02.5 John Dues: So that was the plan. And then the next step in the PDSA cycle is the do and that's just like it sounds. So we ran the experiment for 15 days and then started gathering that data. So what we found was that in two of the cases, transportation challenges were in fact the primary issue, in the third case, it was sort of transportation, but that was exacerbated by a family that was homeless during this period, and then in the fourth cause, or in the fourth case, there was actually some sort of anxiety issues with coming to school. So you can see three very different causes just across four students. So again, if you don't have that picture, then it's very hard to sort of design the right type of interventions 'cause you don't really know what's going on. You have assumptions, right? So I don't think it's rocket science, but the team learn that there are so many layers to this attendance challenge, and even for a single student, there's often multiple factors rather than some single explanatory variable. And so you have to sort of uncover that, and I think the key thing was that holding these empathy interviews, just these four interviews allowed us to challenge some of our initial assumptions. Like maybe a family doesn't value attendance, that didn't seem to be the case, at least with any of these four students who are facing some serious challenges on the home front.

 

0:13:48.6 John Dues: But it wasn't like families didn't value school or having their kids attending school, there're just major obstacles. And so digging deeper allowed us to explore these various causes with the families. Another thing that was interesting is that as we talked with the team about... As the data came in and what they were doing, we also learned that we need a better, more systematic process for intervening with chronically absent students, that's everything from reviewing the data, identifying those chronically absent students early on in the school year, for those that we're required to do something like file truancy for those processes and then monitoring attendance, there's various requirements public schools have on that front. Every school is... They have a system in place, and they have a team in place, and they have a process in place for these different things, but they're all doing it differently, and so there's not a standardized process across our system. And another thing is, some parents didn't even realize that they may have a general idea that the attendance isn't great, but don't... Most parents don't actually realize what is the actual attendance rate of their child, how far off is it from what's considered exceptional or at least okay attendance.

 

0:15:17.6 John Dues: Almost nobody has that. Those numbers at the ready. Another thing that has happened as we studied the data was that there was a really wide variation in terms of the difference in student daily attendance between the period of the intervention and end of the school year up to that point. So there's basically a lot of learning going on with just a very simple four student experiment. So even though the predictions weren't perfect, and one thing with the predictions is, this is Cycle 1, so what should happen over time as we gain knowledge about our system, is that the predictions get closer and closer to what actually happens because we're learning with every PDSA cycle that we're running basically. So the last part is, then you act, so we've done the plan, we've done the do, the study, and the act, and the way I frame this is that you have three As that you can choose from in the Act segment. You can adopt that change that you've tested, you're going to adapt that into the next cycle, or you can say this is not working at all and you can abandon it and just do something else.

 

0:16:34.2 John Dues: Yeah, those are the three options. So what we've actually decided to do, what happens in a lot of early tests, is we're going to adapt Cycle 1 into Cycle 2, and in fact, Cycle 2 has actually already started. But the aim of Cycle 2 now is we're going to increase the extent to which we're involving students and families in the data collection process, and we're going to hold what we call like a... We call this a 5 Whys Empathy interview with each student that we've identified, and then use that to create a plan for a PDSA that's specific to that one student, basically. So it's going to be very hyper-focused and so we're going to collect this data for two weeks, we recognize that doing this intensive of a process with the entire school or the entire group of students that are chronic absent probably isn't possible, but what we're doing is learning so much from this, that seemed like we're going to take another step to learn more and work with the individual student to set up the next round of interventions.

 

0:17:49.4 Andrew Stotz: And what are you guys expecting for an outcome? You know, talking about prediction? I don't know. Should we think about where are you going to be in one year or two years, three years?

 

0:18:00.7 John Dues: Oh no. It's very closely tied with the PDSA. So if it's a two-week PDSA, then we're actually saying, what is that the average daily attendance going to be for that two-week period?

 

0:18:11.7 Andrew Stotz: Yep. Yep.

 

0:18:14.3 John Dues: I mean. It's very tightly closed. Yeah.

 

0:18:14.8 Andrew Stotz: What I'm saying though, let's just take the attendance levels that we've seen in the chart, let's just talk about annual and let's say, "Okay, one year from now, two years from now."

 

0:18:23.7 John Dues: Oh yeah. That's right. Oh I see.

 

0:18:26.7 Andrew Stotz: Are you... Is it right to make a prediction about where you think you would be or is that not the right way to do it?

 

0:18:32.0 John Dues: I think it's too early in the process to make... I see what you're saying now you're talking about the actual... That overall system measurement. Yeah, I think it's too early to make a prediction on that, if you were holding a gun to my head and making me put money down, my prediction would be right in line with what it is this year, basically. I would think it... Because it's a stable system. Those last four years, all bounce around 50-55%, like you said. So my bet would be on 50-55%. Because...

 

0:19:11.2 Andrew Stotz: And what would... How would things change for you or the school? Let's just imagine hypothetically... I'm going to push back a little bit here and just get some thinking, but from a hypothetical perspective, let's say a new school opened up and their number one focus was chronic absenteeism, and they decided that the most important thing for them is to solve that problem. And they had been doing it for a while in other locations, and now they've come in, now you're competing with them. They're an option for some people, and they're just the place for others, and let's just say that they have optimized for chronic absenteeism and it's down to 10% at their school. Does that change how you think about what you're doing? Again, it's hypothetical, but I'm just curious. How does that...

 

0:20:01.3 John Dues: No. Not me, because this is what our system produces right now, so... Yeah, I think I would say I don't have enough information to make a prediction about what the ultimate outcome measure will be.

 

0:20:19.4 Andrew Stotz: And I guess you could say this is what our system produces in this area based upon what we see as important, right? Like this is...

 

0:20:32.4 John Dues: Yeah. That's fair.

 

0:20:33.2 Andrew Stotz: And there may be another area that you think is very important in that those numbers in that area may be very, very different from that, but another... Go ahead.

 

0:20:43.7 John Dues: Yeah, well, I was going to say, so in this attendance team, there's... The empathy interview is going on with these four students, and then in the Act phase, we also said we're launching an effort to make the intervention process more consistent across all of our schools. And we're starting by understanding the process that's in place right now, but this team is not the only team pulling levers that ultimately could impact attendance. I think they're pulling important levers, but there are other improvement teams across the school system, for example, I think I mentioned this maybe a few episodes ago, transportation. Transportation did come up like we thought it would in three of the four interviews. So, and we have mentioned how poor the bussing has been in Columbus for the last several school years, but especially last year and this year. And so we're working to see can we do something significantly different next year on the transportation front.

 

0:22:00.1 John Dues: And I think if we can pull that lever, that also... But that would be a change to the system, like a very significant change. Now, if that went through, I would be much more confident about making a prediction about improvement on the attendance front next year, so.

 

0:22:17.9 Andrew Stotz: If I look...

 

0:22:19.2 John Dues: It's not going to solve our problems though. Yep.

 

0:22:21.2 Andrew Stotz: Yeah. If I look at my roasting business for coffee, if I replace my pretty much manual roasting machine with a fully automated high-tech one, it's going to have a massive improvement in variability. Invariability is going to reduce way beyond what our current system can produce and it will happen in a day, right? When that happens, it'll happen in a day.

 

0:22:47.4 John Dues: Yeah.

 

0:22:52.9 Andrew Stotz: And being late for class is a big bag-a-boo of teachers here in Thailand, particularly at universities where I go to, and it's a problem and nobody likes it and students come in late and all that. But I solved that problem with just the twist of my finger, one twist of my fingers, and I solved it. What was that twist? I locked the door. And then as the students were outside waiting to come inside, I would eventually go out and I'd say, "Look, it's important to me that you're on time. I'm going to lock this door. If you can't make it... You got to figure out how to make it." I know you got two hours of traffic and you're coming in from abroad, or you're coming in from outside of the city, I know that your parents don't have the money to pay for a car for you and you got to take the subway or you've got... I know. Everybody's got their circumstances, but you're making an effort to get here, I want you to get here on time.

 

0:23:52.4 Andrew Stotz: The next class that I have, everybody's on time. So one of the questions I have, and this is, again, push back is, some people may look at this and go, "Oh. Come on. All this work. Why don't we just massively prioritize and focus." Let's just say that... Let's just say, I don't know what the answer is, but let's just say that the principle of the school, all the teachers and all the students gather out in the front area at the time that you're supposed to be at school, and there's a band playing. Everybody's cheering. We're getting ready. Whatever that thing is. I remember a boss I had that used to have a stand-up meeting every single morning, and you didn't miss it. And so for some people who are listening, they may think, "Oh, come on, John, you're going through all this stuff and it's not going to improve. Why aren't you just taking more aggressive action right now."

 

0:24:48.9 John Dues: Well, I didn't say I was going to improve it, I just said I wasn't going to make a prediction.

 

0:24:51.9 Andrew Stotz: Yes. Yes. Sorry. I didn't mean...

 

0:24:53.8 John Dues: So yeah. What would I say? I think when you have... So the person that is in charge of this project, for example, I don't know that anybody's ever gotten better results while a principal of a school in Ohio with the challenges that he faced. So this is not people that aren't driven to get extraordinary results. This is a multi-faceted problem that is incredibly tough to improve, and when I have the view point of sitting down with the students and hearing what is... Keep the obstacles are... It is just a very hard thing, and I'd say one of the reasons we're pulling that transportation lever early, or at least attempting to is because that's such a big part. Again, that's not going to solve everything. But like I said, if we were able to pull that lever and it's a big if because we get no funding for transportation. All the funding flows through the school district, so that's a massive obstacle. Millions of dollars that we aren't allocated to do this service, someone else is. So right now we don't have control over it, and so those are the types of obstacles, we can't just buy buses, for example. Because...

 

0:26:36.1 Andrew Stotz: Yeah. And it's like you got 50 problems that you're trying to...

 

0:26:41.7 John Dues: 50 problems.

 

0:26:41.7 Andrew Stotz: And 50 more constraints that you're operating within.

 

0:26:45.9 John Dues: Yeah. And so we were attacking it from multiple angles, so we were attacking it on the attendance front with kids, at the State House with funding, with trying to see if we can set up our own transportation system even without that funding. So there's many, many levers being attempted, but they are not quick and easy, simple, so.

 

0:27:12.8 Andrew Stotz: And for a listener who's listening to this, who may not be the CEO of a company, let's say who's got resources he can allocate or she can allocate, they also may be in a situation like, "This is all I can impact. I can impact this area, but I have to be realistic about what resources I have."

 

0:27:33.6 John Dues: Yeah, and I think one of the things we're doing too, we don't have rose colored glasses on, we're saying, even if we did fix this transportation system, and that's a big if again. That doesn't mean that the chronic absenteeism problem is going to be solved. Maybe it's significantly better, maybe it goes down to 30%, 35%, if we had a good transportation system, but that still means... And even before transportation was such an issue, even before the pandemic, these rates were still... The quarter of the kids who are so chronically absent, that's way too high, that's way too high. So we recognize that, but these are very, very thorny problems to try to improve.

 

0:28:15.0 Andrew Stotz: Yeah. Well, and also you're trying to make lasting change too, so.

 

0:28:18.6 John Dues: Lasting change. Yeah.

 

0:28:19.6 Andrew Stotz: Yep.

 

0:28:19.7 John Dues: Yep. Yeah.

 

0:28:22.1 Andrew Stotz: Okay. Great. So I'm going to wrap it up there. And thanks...

 

0:28:23.3 John Dues: Yeah. Absolutely.

 

0:28:24.6 Andrew Stotz: And thank you for, on behalf of the Deming Institute, and I want to encourage all the listeners out there to follow what John's doing and what he's talking about, and of course, go to Deming.org to continue your journey. You can get his book, Win-Win, W. Edwards Deming, The System of Profound Knowledge and the Science of Improving Schools on amazon.com. And this is your host, Andrew Stotz, and I'll leave you with one of my favorite quotes from Dr. Deming, people are entitled to joy in work.