Addressing Assessment of Untaught Material

What if learning could feel like a team sport instead of a pressure test? Lyle "Lee" Jenkins, PhD., a longtime educator, shares how a chance encounter led him to a Deming conference specifically for educators in 1992, which transformed his thinking. Deming emphasized defining learning outcomes, rejecting numerical goals, and avoiding ranking. Lee explains how Deming methods prevent "cram and forget", celebrate small wins, and rekindle students' natural love of learning. (Lee shared a powerpoint during the episode, which you can find on our website.)

TRANSCRIPT

0:00:02.3 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we continue our journey into the teachings of Dr. W. Edwards Deming. Today I'm talking with Lee Jenkins, who is a career educator in public school, ending his career as a school district superintendent. It was as a superintendent that he was introduced to the teachings of Dr. Deming, and he has been applying it to his life and work since then. In his business, Crazy Simple Education, he publishes books and schedules speaking engagements. Lee, how you doing?

0:00:38.4 Lee Jenkins: I am doing just great, Andrew. Yeah, this has been fun to put together. And just to highlight, I haven't done this before, just to highlight just simply what Deming taught. We've obviously, over the years added other things, but today we're just talking about what did he teach, just the pure form of it and our implementation of that.

0:01:01.6 Andrew Stotz: Yeah. And I think you and I have already met once and gone through this. It's pretty interesting, you know, I think what I enjoyed about our discussion, truthfully, what I liked, was your energy and the energy about the teachings of Deming and how we can apply that. And so I'm looking forward to seeing you bring that to the audience. Now, for those that are listening, we're going to have... Lee's got a PowerPoint and a presentation he's going to share, but we'll walk you through it. It's not like it's full of very complicated things. So, Lee, why don't you take us through a little bit about what you've prepared here?

0:01:38.3 Lee Jenkins: Okay, I can do that, Andrew. I was like anybody else as a school superintendent. I went to a meeting of the Association of School Administrators. I can't even tell you what city or state it was in, but I was there. And while I was in the hallway between sessions, Lew Rhodes, who worked for AASA, he came up and he said, "Lee, I think you'll enjoy this next session." And that's why I've called this, One-Minute Invite That Changed My Life. So I went in and no idea, I just liked Lew. I trusted him. And it was David Langford's an administrator. And that's how I was introduced to Deming and spent a lot of time after that, reading everything I could get my hands on and absorbed it. And I knew that he was correct in how organizations are operating. And so that intrigued me a great deal. But it was the same information that he shared with all organizations. I just took them and applied them to education. But then two years later, in 1992, American Association of School Administrators, under... With Lew Rhodes' leadership, sponsored a Deming conference. So I went to Washington, DC in January that year to hear him speak.

0:03:20.2 Lee Jenkins: We were there four days. He was assisted and was a part of it for two days. And for two days it was him on stage, the red beads, you know, all the things that listeners know about with Dr. Deming. And I would say that the first part of it was the things you would normally expect to hear. Now, understand, the audience here was educators. And I know there were educators sprinkled in his audiences in his whole speaking career. I know that. I wasn't one of them, but I know that. This was one that was specifically for educators. And nobody's told me any other time when he spoke to educators as the audience. So, but just things he'd say that we've all heard.

0:04:13.7 Lee Jenkins: Best efforts are not enough, you have to have knowledge, you have to have theory. He said too, you can't delegate quality. And I had school superintendents doing that all the time. You ask them about, anything about teaching or learning, they say, oh, no, I'm not involved in teaching and learning. I have an assistant superintendent for instruction. In other words, they've delegated quality. Deming talked about wasting time and wasting money in all organizations, and certainly schools are good at that. I'm going to talk at the end of this, how I took it onto one other point which is similar to what he's talked about also. The losses of the current system. He said in one place that, for 50 years... Now, he said this in the '90s, but for 50 years, America has been asking for better education without a definition of what better education is. And...

0:05:10.5 Andrew Stotz: That reminds me of talking to Bill Scherkenbach, who showed a picture of him, Dr. Deming, in the old days at an event of national teachers, and he said they really couldn't come up with a conclusion about what was the aim. [laughter]

0:05:25.9 Lee Jenkins: Yes, right. It's... Yeah, okay. And then he described fear, brings about wrong figures. So what did our government do? No Child Left Behind, which says, you increase your reading scores or your math scores or we're going to fire you. Well, then you get wrong numbers. That's what he predicted, that numerical goals are a failure. I had a discussion with a pastor several years ago and he said, "Our goal is to have 2,000 people in attendance on Easter Sunday." I said, "Okay, what's the best we've had so far?" "It was around 1800." "Okay, what happens if we have 1900 on Easter Sunday, the best ever? What do we do?" Well, it kind of caused him to think, which is my purpose. It wasn't to be critical, it was to get him to think. You could do your best ever but call yourself a failure because you didn't meet this artificial number. And I can hear Deming talking about just pulling the number out of the air. And that ranking is a failure. We rank and rank and rank in schools. I've got a granddaughter in first grade. School has just started. She's student of the month in her class, which means there's 19 failures of the month. I mean, Deming, it's just sad to see that it's still going on. But then Dr. Deming, I don't think it was in... It wasn't in his PowerPoint. Not even a PowerPoint. We had transparencies.

0:07:12.0 Andrew Stotz: Acetates.

0:07:12.6 Lee Jenkins: It wasn't in his transparencies. It wasn't in the handouts. But it's like he went on this little tangent and that's what has captivated my career, his tangent. And it was Dr. Deming, the statistician, talking about the classroom. So I'm going to go through what he said, just as he said, point by point. He said, number one, tell the students what they will learn this year. Now, when I share this with people, they say, oh, yeah, our college professors had syllabuses. I said, no, no, a syllabus is what the professor is going to teach. Dr. Deming talked about, what are they going to learn? They're two different things. What are you going to learn? And you give it to them. And we've done this pre-K, kindergarten all the way to grade 12 and a little bit of work at universities.

0:08:14.6 Andrew Stotz: And how detailed do you go on that? I see you're showing concept one to concept 19. Is it, you know, this is everything you're going to learn, or this is generally what you're going to learn?

0:08:26.5 Lee Jenkins: Well, this is a partial list. So it's the essential.

0:08:31.6 Andrew Stotz: Yep.

0:08:32.6 Lee Jenkins: I tell people, put down what's essential. Do not put trivia on the list. Now, of course you teach trivia. It's interesting, it's fun, but they're not accountable for it. And so it's what students have been asking for for years. What am I supposed to learn this year? I don't know how to study for the exam. I don't know what's important. I was at a... Doing a seminar for teachers in Missouri. And I said, "I wasn't a good test taker in college. Were some of you?" And a lady raised her hand and said, "Oh yeah, I was really good at it." I said, "How did it work?" She said, "Well, I was in a study committee and by design, half of our time was sharing our insights as we psyched out the professor. And then once we agreed on what was important and the personality of that professor, then we studied that." That's nonsense. Here's Dr. Deming saying, just tell them what you want them to learn, it's so simple.

0:09:47.0 Andrew Stotz: In the world of teaching, we often talk about learning outcome statements at the beginning of a lecture.

0:09:55.6 Lee Jenkins: Yeah.

0:09:56.5 Andrew Stotz: And I know, for instance, with CFA for Chartered Financial Analysts, they have very clear learning outcome statements and then they have a whole section that they teach and it's self study. And then you take an exam. Is that... Is learning outcome statement the same thing or is this something different?

0:10:13.0 Lee Jenkins: I would say it's the same. It's very, very close. It's same in general terms. Exactly. We're not talking about how it's going to be taught, only that it's going to be learned. Okay, the next thing Dr. Deming said to do... And by the way, before we leave, make sure this is a partial list. If I put the whole year's list on there, it's so small nobody could read it on the screen. Okay, next he said, give the students an exam every week on a random sample from the whole course. Said if, for example, you had a 100 concepts on your list, they would take a quiz on 10 of them each week, randomly selected.

0:11:02.6 Andrew Stotz: This is so mind blowing. Go ahead, keep going.

0:11:07.7 Lee Jenkins: Yes, because... So what do we do now in schools? We do cram, get a grade, forget. That's the most common thing in American education. Cram, get a grade, forget. Have a friend in college. He said, "Lee, I've looked at your website. I have a little bit of an idea of what you do. You don't know this about me, but I never studied the night before an exam in college, ever." "Oh, really? What'd you do, Larry?" He said, "Well, I set the alarm for 4 o'clock in the morning. I studied the morning before the exam." I said, "Why is that?" "I couldn't remember it overnight. So I did well in college. I got the grades on the exam and by noon it was gone. But I got through. That was my system." I was at my annual dermatology exam and the medical doctor said, "What do you do?"

0:12:20.7 Lee Jenkins: I said, "Well, actually I get on airplanes and I give speeches." "Ah, who do you give them to?" "Well, teachers and administrators." "But what do you tell them?" "I tell them how to set up a system where it's impossible to cram and forget, you just have to learn." She said, "Oh, that's interesting. That's what I did all the way through medical school." And I'm thinking, here I am with somebody who crammed and forgot all the way through. So I checked with an MD on my next plane flight who I happened to be sitting next to one. I told him the story. He said, "Yeah, that's how it works." I said, "Well, when do you learn?" "Residency." So Dr. Deming didn't talk about cram, forget. But the side effect was, when the students don't know what's coming on the Friday exam, they'll say to me, I just have to learn. There's no other choice. You just have to learn.

0:13:25.8 Andrew Stotz: Right. And then you talk about the... You're talking about the random sample size is roughly the square root of total concept list. I'm thinking about a 15 hour course that I teach and there's 25 concepts that I'm teaching. So a random sample would be 5 of those 25, give them that test. And then the idea here is that we're testing their understanding of that material. And in the beginning, let's just say that random, in the beginning, I haven't taught anything. So they have five questions and on average, let's say they get one right in the beginning because...

0:14:05.2 Lee Jenkins: You'd be lucky if you got an average of one. Yes.

0:14:07.8 Andrew Stotz: So we have evidence that they don't know the topic.

0:14:10.9 Lee Jenkins: Right.

0:14:11.6 Andrew Stotz: And then as we... Let's say we have five weeks and each week we go through, then in theory, if we've taught right and they've learned right, that they would be able to answer all five of those randomly selected questions on the fifth week?

0:14:29.3 Lee Jenkins: That's what you're after. You want them to not have to study, but whatever five is pulled out, they would get it. And if you're teaching a five week course, you might give 10 quizzes during the time, one at the beginning and one at the end of each class. So that because the random, you want them to have questions come up more than once, you want them to have the same question come up. Because that's part of the joy. Oh, we've had that, it's been taught or I've seen that before and it's not 25 questions, it's 25 concepts. Because you can ask it a multitude of different ways to see if they have the concept.

0:15:09.3 Andrew Stotz: And for teachers nowadays, or administrators, they're going to say, what's the point of giving quizzes for topics you haven't taught?

0:15:22.7 Lee Jenkins: That is the most common thing I've been told. Okay. And teachers who have done this for a number of years, sometimes 10, they will say that is the most powerful part of the whole process. Think of it as the synonym for what Dr. Deming taught as review preview. People are used to previews of movies and TV shows and all kinds of previews. And that's what it is. It's a preview. It's not graded. You know, the quizzes aren't graded. That is not fair.

0:16:00.9 Andrew Stotz: You mean they just don't count... They don't count as a grade for the students?

0:16:05.4 Lee Jenkins: Don't count as a... They're scored.

0:16:07.0 Andrew Stotz: Yeah.

0:16:07.3 Lee Jenkins: They're scored...

0:16:08.6 Andrew Stotz: They're scored.

0:16:08.7 Lee Jenkins: But they're not ABCDF on it. Yeah.

0:16:10.3 Andrew Stotz: Yep.

0:16:11.1 Lee Jenkins: It's just a number, correct. Yes. And so like a geography teacher, excuse me, science teacher, said, "You can't believe what happened to me last Friday. I said to the students, on Monday, we're going to start a unit on rocks. And these are middle school students. And they all applauded." He said, "I've never had students applaud about rocks before." Why? Because it keeps coming up on the quizzes and they want to know. It does that. And then when the students get things right that the teacher hasn't taught yet, then they get, oh, they're really happy. I outfoxed the teacher. I know that.

0:16:57.8 Andrew Stotz: Yeah. You can also imagine it would be interesting if you gave a test and the score was, you know, a four on average out of five, let's say, right at the beginning of the class, you think, wait a minute, they already know this stuff. How did they learn that? Where did they learn that? What am I doing in this class?

0:17:15.1 Lee Jenkins: And see, and one of the things we have to get our heads around is, it doesn't matter how they learn it. The question is, did they learn it? I mean, with AI out, okay, they can... They could do AI... They could find out on their own. They can ask their parents. I mean, there's books, there's the Internet. It doesn't matter. Did they learn it?

0:17:40.4 Andrew Stotz: Yep. Okay, this is great.

[overlapping conversation]

0:17:42.5 Lee Jenkins: So then Dr. Deming said, if you've got 100 concepts, then you'd have 10... It's what he said. You'd be 10 questions a week. So that was in January and in November, I wrote him a letter and we had teachers in the school district already doing this. "Thank you for your kind letter and for the 100 sided die." I had just seen that and they're on Amazon. You can buy a die that's 100 sides. It's like the size of a golf ball. He said "it's exciting. Thank you also for the charts, which I've looked at with interest. I wish for you all good things and remain with blessed greetings. Sincerely yours, W. Edwards Deming."

0:18:29.3 Andrew Stotz: That's cool. And that 100 sided die, that was just saying, if you had 100 concepts, just roll the die and pick whatever ones that land... The 10 that lands on it.

0:18:43.1 Lee Jenkins: Right. Now, I've discouraged over times people landing on 100 because you want essential. So to get to 100, you either have to add trivia to get to 100 or you have to take away essential to get down to 100. So I want people to put down what is it that's essential for their kids to know and when they see them 10 years from now, they still know it.

0:19:10.0 Andrew Stotz: Yep. Okay. So, let's not... We're not going to fixate on 100 is what you're saying.

0:19:14.6 Lee Jenkins: Don't fixate on the 100. But I'm telling what Dr. Deming said as an example. Yeah. And what we did. Okay. Then he said create a scatter diagram. This is not a scatter plot, it's a scatter diagram. So if you look at the bottom left, you can see that... And let me find here, if I can just pointer options. Let's get this. Okay, if you look right here, this is Quiz 1, Quiz 2, Quiz 3. Over time...

0:19:49.4 Andrew Stotz: Okay. So the... Just for the listeners, we're seeing a document that's up here with a 14 quizzes across the bottom. Yep. And then on the Y-axis...

0:20:03.1 Lee Jenkins: And the Y-axis is from 0 to 10.

0:20:06.5 Andrew Stotz: And that's the quiz questions.

0:20:09.8 Lee Jenkins: No, it's... They were asked 10 questions. Yes.

0:20:12.0 Andrew Stotz: Okay. So in this case we can see...

[overlapping conversation]

0:20:12.7 Lee Jenkins: The question number...

0:20:12.7 Andrew Stotz: And then those questions were randomly selected. And then they were put into a quiz format of 10 quizzes, quiz questions. And here we can see, for instance, question number two, four people, I'm assuming, got it right.

0:20:29.8 Lee Jenkins: On quest... This is... On quiz two...

0:20:31.0 Andrew Stotz: Quiz number one, let's say quiz number one, question number two.

0:20:35.7 Lee Jenkins: Quiz one, nobody... One person got zero right. One person got one right. Four people got two right.

0:20:41.7 Andrew Stotz: Okay. Okay. I see.

0:20:42.8 Lee Jenkins: One person got three. Okay?

0:20:44.8 Andrew Stotz: Yep.

0:20:45.8 Lee Jenkins: These are people for quiz one.

0:20:49.1 Andrew Stotz: Okay.

0:20:50.3 Lee Jenkins: Then this is quiz two. And then this is quiz three. Generally one each week. We've landed on seven times a quarter, because think snow days come up, things happen.

0:21:09.0 Andrew Stotz: Yep.

0:21:09.5 Lee Jenkins: But so seven out of the nine weeks works. So this is the quiz for a semester.

0:21:16.8 Andrew Stotz: Okay.

0:21:17.6 Lee Jenkins: And the end, at the 14th week, a 14th quiz, I mean, you've got one, two, three, four, five, six. We've got all 10 right. You got four of them with nine, et cetera. That's your Scatter diagram.

0:21:32.2 Andrew Stotz: Okay.

0:21:33.4 Lee Jenkins: Okay. Then he said, do that. Then he said, which I've heard nobody else ever say, add up the total for the whole class. That is unbelievable. Think about it. When an athletic team wins, the players and the coaches celebrate together. In schools, when the final's over, the students celebrate and they do not invite the teacher. Here, every time they are tracking their work, this is quiz one, quiz two, quiz three, four, five, six, seven. It's an interesting one. Somebody put this chart up on a bulletin board, put push pins up and connected with rubber bands.

0:22:24.5 Andrew Stotz: Right.

0:22:25.8 Lee Jenkins: Okay. Here's another one where they're learning that the United States states, they have a blank map of the United States. An arrow points to one of the states. They have to write down what state that is. And there they are. And this shows the progress over 18 quizzes. And you can see it going up and up and up. And here's one, two, three, four, five, six, seven, eight times, where... And maybe there's another one. But you're... I'm covered... Oh, there is another one. There's nine times that the class did better than ever before as a team of learners. And they celebrate together, the teachers and the students together.

0:23:16.1 Andrew Stotz: Right.

0:23:16.7 Lee Jenkins: Look what we did. Then here's another one. This one on the left is from Australia. And I don't know what subject it was. There's no information. But I know that they went out and took a picture of it with one of the students holding it because they were so excited they'd hit the 200 mark after having started out at 65.

0:23:41.3 Andrew Stotz: Yeah. And for the listeners, we're... Basically Lee's showing different run charts of the number correct, starting from quiz number one all the way through to the final quizzes. And the number is going up and to the right showing that the process of learning is working.

0:24:03.4 Lee Jenkins: Yes. And this one here is spelling. We know that spelling doesn't... Spelling tests don't work. It starts in first grade. It's the classic cram on Thursday night if your mom makes you, take the test on Friday, forget on Saturday. So here is a classroom with 400 spelling words for the year. They're all put in a bucket and 20 are pulled out each... 20 are pulled out each quiz at random. And you can see they're learning the words. Now, sometimes people think that we teach at random. You don't teach at random. You teach logically.

0:24:40.1 Andrew Stotz: Right.

0:24:41.2 Lee Jenkins: But the random is giving you accurate information. Are the students actually learning it and not just playing the game. And here's a... You want students to do the work as much as possible. They're your student. That is when you see the coloring and the art, the creativity. Yeah, that's... You want to hand that over to kids to do as soon as you can. And here's one. A French class out of Canada. This is a Spanish class, a third year Spanish class. And the teacher has written that ABC, ABC, ABC, because the teacher had three different quizzes all for the same concepts. So they got quiz A, one week. Quiz B the next time. Quiz C the next time. Whatever, random numbers, but then she had three different complete sets of questions for each of the concepts.

0:25:37.0 Andrew Stotz: Right.

0:25:39.6 Lee Jenkins: Oh, I love this one here. The class had 69 correct, then 108, then 128 right as a class. Then they slumped. One, two, three, four, five, six weeks they slumped and they ended up 129 correct as a class. One more than ever before. The kids are thrilled. If you don't count it up, you'll never know that as a teacher. You'll never know it.

0:26:07.3 Andrew Stotz: And you wouldn't know your progress relative to your past class.

0:26:11.2 Lee Jenkins: You would not.

0:26:12.0 Andrew Stotz: Yep.

0:26:12.8 Lee Jenkins: And so I can't tell you how many teachers have told me, when they have a... The class has an all time best by one or two, a student in the class who's been struggling will stand up and do a chest pump and say, it was me.

0:26:27.6 Andrew Stotz: Yeah.

0:26:28.8 Lee Jenkins: If it hadn't been for my correct questions, which were few in number, but hadn't been for mine, the class wouldn't be celebrating. Yeah, we all understand that, if you're a poor athlete, you're on the basketball team and you're on the bench and the coach decides to put you in for a little bit. The other team fouls you because they know you're not a good athlete. But you make the free throw and the team wins by one.

0:26:57.3 Andrew Stotz: Right.

0:26:57.6 Lee Jenkins: You don't hang your head and say, we only won by one. No, you and everybody knows you're the one that made the point that counted, yeah, it's the same thing. And I've wrote this, it's so important. But sports teams celebrate together, coach and athletes, with class run charts, teachers and students celebrate together. So since 1992, we have subtracted nothing from Dr. Deming, what he taught. We've added some clever additions. The little dots on there that say all time best, that's an addition. We changed it from every week to almost every week. And if we have a chance to do another podcast, I will focus on all the things we've added that are so creative, that have come mainly from students. But what Dr. Deming said, and I'm estimating it was three to five minutes, that he shared and they went back to his normal program and it just impacted me. I couldn't believe it.

0:28:15.7 Andrew Stotz: Right.

0:28:17.9 Lee Jenkins: On the website, Crazy Simple Education, there are free blank graphs. So if anybody's interested in what I'm talking about, there's... If you're... And you'd have to look at, if I'm adding... If I'm asking five questions a week, then there's question... There's graphs for that. If I'm asking 20, there's... They're all there. And other things.

0:28:36.6 Andrew Stotz: Yep.

0:28:39.6 Lee Jenkins: So there's kind of just my little bit of the bio, but it's already been shared. And then on the website, if anybody's interested after over 25 years, what would be the most detailed information of Dr. Deming it's in this book. But you're going to get that information in the future anyway. But I'm just saying, it is there.

0:29:10.9 Andrew Stotz: And just for the viewers, that book, go back to the book for a second. For the listeners, it's called the Essential Navigation Tool for Creating Math Experts, Numbers, Logic, Measurement, Geometry.

0:29:24.0 Lee Jenkins: It has the actual quizzes for grade five, the 28 quizzes for the year. They're there.

0:29:31.2 Andrew Stotz: Right. Right. Amazing.

0:29:33.0 Lee Jenkins: It is superbly put together. Each of the concepts in grade five is assessed seven times. Each of the grade four concepts are assessed twice during the school year. And each of the grade three concepts are assessed once during the year.

0:29:53.5 Andrew Stotz: Yep.

0:29:54.0 Lee Jenkins: So you don't have to waste the first month or so going over last year. You just start the new content and the review is built in.

0:30:02.4 Andrew Stotz: Yep. Okay. And for the listeners and the viewers, we're not trying to sell this stuff. What we're trying to do is show it as an example of the things that you're doing, which is great.

0:30:12.6 Lee Jenkins: Yes. Yeah. It just shows what can be done with the simple concepts.

0:30:18.5 Andrew Stotz: Yeah.

0:30:18.8 Lee Jenkins: And this is one example. Yes. And so then Dr. Deming talked about waste. And he also said that graphs have to be long and narrow. So here's my long and narrow graph on waste. I asked 3,000 teachers, five different states, just what grade level do you teach and what percentage of your kids love school? Okay, well, kindergarten teachers said 95% of their kids love school. First grade said 90%, second grade said 82% love school. And it goes down every year. It gets fewer and fewer kids love being in school until we get a low of 37% love school in grade nine. It ticks up slightly in grades 10, 11, and 12. But I show this to people, the most common answer I get is, well, of course it went up in grade 10, 11, and 12. I dropped out of high school. They didn't count me.

0:31:25.9 Andrew Stotz: Yeah, yeah.

0:31:28.6 Lee Jenkins: So, but, so the biggest waste in education is the love of learning kids bring to kindergarten. Much more damage caused by that than wasting time and money. That the kids have all the motivation they need for life in that five-year-old body. It's not our job to motivate...

0:31:52.4 Andrew Stotz: And then we flush it out of them.

0:31:52.4 Lee Jenkins: Yeah. It's not our job to motivate them. It's the job to maintain it. So I'll tell you a story of a good friend that I worked with from the very beginning. I mentioned that when I had the note that went off to Dr. Deming. And after we'd just gotten started, he's still teaching grade eight science. He has five periods of science. He says every year, the first day of school, three, four, five eighth graders come to him each period. And they say, "Just so you know, Mr. Burgard, I hate science." So he says to them, "Oh, that's interesting. How long have you hated science?" The kids say the same thing every time, "I always hated science." He says, "You know, actually, that's not true. You loved everything in kindergarten. Tell me your story." And they tell the story. Well, I was in grade three or I was in grade five, whatever, they tell their story. He says, "Okay, here's the deal this year, I'm not going to motivate you to learn science. What I am going to do is to try to put you back the way you used to be. We're going to put you back with the mind of a kindergartner loving learning. That's what we're going to do." Because they... Everybody has stories on what happened to them.

0:33:23.4 Andrew Stotz: Yep.

0:33:24.3 Lee Jenkins: So I would conclude this part by saying, I am forever grateful to Dr. Deming. My younger son went to the Deming Scholars Program with Joyce Orsini and he graduated. I got to meet both Diana and Judy Cahill, and they were helpful. Kevin just been helpful to me. Kevin Cahill, the grandson, David Langford, I met with him in-person probably 20 times. All encouraging. Jake Rodgers now is the reason why we're here. And of course you, Andrew. So there's so many people to be grateful to that have encouraged me along this journey, in addition to several thousand teachers who send me their stories and their pictures of their graphs, thanks.

0:34:14.1 Andrew Stotz: Fantastic. That's quite a story. And I just love those lessons that you've gone through. I'm going to stop. Is it okay if I stop sharing the screen? I'm going to do that myself here. Is that okay?

0:34:27.9 Lee Jenkins: Yes.

0:34:28.4 Andrew Stotz: Okay, hold on. Don't do anything there. Okay, now I see you, you and me. I want to wrap up because I think that was a great presentation. A lot of things that I'm thinking about myself. But I did have one question for you that I... I'm not sure what to do. One of the things that I've found with teaching is that sometimes my students, they have a hard time focusing. And so when I tell them, okay, you need to read chapters one, two and three before we meet the next time, let's say short chapters. And then they find it's hard for them to stay, they're like, ah, I'll do it later. So they really haven't covered the material. Now, if I give them, if I say, you need to read chapters one, two and three, and I'm going to have a short quiz on chapters one, two and three, and I'm going to give you quizzes every time that we meet, not as an objective to score your work, but as an objective to help you keep focused. And then I do that, let's say five times, and then I take the two best scores and I drop the rest, so, it shows that they did it. And I find that my students, they definitely do... They stay up on their work with it. So my question is, how do I incorporate this, which is really an assessment of the learning in the class with that, or do I need to drop what I'm doing with my quizzes?

0:36:00.6 Lee Jenkins: Okay, we're really talking about the difference between them intrinsically wanting to learn it and being pressured to learn it.

0:36:13.8 Andrew Stotz: Yep.

0:36:14.0 Lee Jenkins: In a sense. Okay? Now, one of the parts I did not share that could be for future. But the students do graph their own work. Dr. Deming didn't talk about that, but that was... I just focused on what he taught. They graph their own work. And then there's the graph for the whole class. They want to know if they have a personal best. They care about that at all grade levels.

0:36:41.8 Andrew Stotz: Yep.

0:36:42.4 Lee Jenkins: When I... My work is with teachers and if it's a two-day seminar, there's three quizzes, day one and three, and three more quiz, two, day two. There's... You see them, high five. They're teachers. They got... They did better than ever before. Other people are congratulating them. They're so happy. And then at the table where they... Because they usually sit about six or eight at a table, they can see their table did better. There's a chart up on the wall, that's everybody in the room. It might be 200. And altogether we did better than ever before. They care about that. And so kids...

0:37:29.6 Andrew Stotz: Okay so from that, do I take from that drop the quiz that I'm doing and replace it with what you're talking about and get them excited about that and then they'll do their work naturally.

0:37:41.3 Lee Jenkins: Because they don't want to let the team down.

0:37:45.7 Andrew Stotz: Right.

0:37:46.1 Lee Jenkins: Okay?

0:37:46.5 Andrew Stotz: Okay.

0:37:47.7 Lee Jenkins: One of Dr. Deming's story for business was, a businessman came, listened to him and he had salespeople on commission. He went back after hearing Dr. Deming and he said, I'm not going to pay everybody their individual commissions anymore. We're going to put all the commissions in a bucket and everybody gets the same amount. So what happened? The best salesperson quit and the company sales went up because everybody wanted to help the team. They couldn't... They didn't want to be the freeloader. They wanted to contribute. But when you think, oh, that person always gets the free trip to Hawaii. I'll never get that. It's not motivating. It's demotivating.

0:38:37.7 Andrew Stotz: Right.

0:38:38.4 Lee Jenkins: And so they want to help. My only time that I know about a good experience in college, was a professor teaching masters students. And he taught the same class on Monday night and Tuesday night. They were doing research methods from all departments on campus. He gave the quiz on Monday night and then the same goes on Tuesday night. And students, they're taking night classes. They don't come every time, things happen in their lives. So it used to be if a student said, I can't come next Tuesday night, they just wouldn't come. Now they say I can't come next Tuesday night, is it okay if I come on Monday, if I do that and take the quiz, will you put my score on the Tuesday night group? Because they don't want to let their team down. Here they are in their 30s and 40s and 50s, getting their master's degree and they care about... So it's... And then something else we haven't talked about, that we have graphs for the school. It's the whole... It's the school-wide graph. And every teacher has to turn in the total for their classroom for whatever subject they're doing it with by a certain time. And then there's a graph in the hallway for the whole school. Teachers you're not going around the clipboard and inspecting the teachers to make sure they turn it in. No, they do turn it in because they want to help... They don't want to let the team down.

0:40:06.4 Andrew Stotz: Right, right. Okay, I got it. All right. Is there anything you want to share in the... In wrapping up?

0:40:16.0 Lee Jenkins: I would say that you will get the question, how can you assess them on things that you haven't taught yet? And the answer is you don't grade... You don't give them a letter grade for it.

0:40:28.6 Andrew Stotz: Yep. So you're assessing their knowledge. You're not scoring that assessment.

0:40:34.3 Lee Jenkins: Yes. Yes. And you will have more fun than you can believe from Dr. Deming's simple concept, no matter what age you're teaching, no matter what subject, you will love it.

0:40:48.8 Andrew Stotz: It's brilliant. It's brilliant because it shows that the teacher cares, that first the teacher says, I know what I want to get you guys to learn in this semester as an example. And it's very clear. And I want to know that you're learning it.

0:41:08.5 Lee Jenkins: Yes. And actually, the hardest part for teachers is to write down on a sheet of paper what they want them to know at the end of the year.

0:41:15.7 Andrew Stotz: Yeah. It forces a lot of structure onto you to have to think ahead of time, what do I... What exactly do I want here? You can't... What you're talking about is really clarifying the learning outcomes.

0:41:28.7 Lee Jenkins: Yes. You can't just say one... Stay one chapter ahead of the kids.

0:41:32.7 Andrew Stotz: Yeah.

0:41:33.2 Lee Jenkins: No, you got to know upfront what it is, and that's hard. That takes time. And you revise it. At the end of the year, you'll say, why did I put that dumb one on there, everybody knows that. Oh, I left off something else that was really important. Why didn't I put that on there? Well, every year you will tweak it, but you're not starting over again, ever.

0:41:54.0 Andrew Stotz: One of the interesting things that I can do is, I have my valuation masterclass, which is an online course, and it's a 12-week course. And I do it, let's say roughly three times a year. So I've got a great data set there that I rep... You know, my repetition is not annual. It's three times a year. I even may do it four. But the point is that, you know, I can just repeat, repeat, repeat, improve, improve, improve, and then show them as...

[overlapping conversation]

0:42:20.1 Lee Jenkins: You can... You got a perfect model.

0:42:21.1 Andrew Stotz: Yeah.

0:42:21.4 Lee Jenkins: Yes, you can.

0:42:22.4 Andrew Stotz: Yeah. No, that's exciting. Okay, well, on behalf of everyone at the Deming Institute, I want to thank you, Lee, for joining us and sharing your Deming journey and just a very tiny interaction with Dr. Deming and what he's teaching, that you've expanded into something to bring that joy in learning. So I really appreciate that. And ladies and gentlemen, this is your host, Andrew Stotz. And I'm going to leave you with one of my favorite quotes from Dr. Deming and I'm going to tweak it a little bit for education because he said, people are entitled to joy in work. And I think today what we're talking about with Lee is that, people are entitled to joy in education.

0:43:04.9 Lee Jenkins: Absolutely. They are entitled to that. Absolutely. Yes. Thank you.