Setting Operational Definitions and Achieving Consistency

Now that we understand Operational Definitions (see Part 12), it's time to figure out how to use them to get the improvements and results you want. Andrew and David talk about examples of useful Operational Definitions and how they can impact all aspects of education (and beyond!)

TRANSCRIPT

0:00:00.0 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we continue our journey into the teachings of Dr. W. Edwards Deming. Today, I'm continuing my discussion with David P. Langford, who has devoted his life to applying Dr. Deming's philosophy to education, and he offers us his practical advice for implementation. Today's topic is how to start setting operational definitions. David, take it away.

 

0:00:27.1 David Langford: So in a previous podcast, we talked a lot about the need for operational definitions, and how that improved systems, and why do you wanna do that. Well, that's part of what Deming talked about with profound knowledge and systems thinking, and it's really important. But the nuts and bolts about how do you begin doing that. Well, the first thing you need to do is to figure out what system am I working on, is the number one thing. What am I trying to improve or design? Am I trying to improve the system of behavior? Am I trying to improve the system of the learning in the classroom? Or whatever it might be. Or maybe you're a principal and all of a sudden you realize, "My teachers, they don't all show up on time to the staff meetings," Right? So staff meeting's supposed to start at 3:40, and we don't ever get a staff meeting going till four o'clock, so... I know when I go places and they're not getting started promptly, etcetera, I'll say things like, "What time do you usually start your eight o'clock meetings?"

 

[laughter]

 

0:01:42.3 DL: Yeah, it usually gets people to go, Oh, yeah... Sometimes they'll say, "Well, usually about nine o'clock, so."

 

0:01:47.9 AS: And, David, I have a story I wanna share that I think can kind of lead into this, is that I was involved with a master's in marketing program here, about 75 students every time. And then I'm also involved with a MBA, an executive MBA program with another university. And one of the things that's interesting is the master's in marketing program, 75 students, so these classes are big, pretty big, 75 students, not one of them was late, ever. All 75 were in the classroom, door closed, when it was time to start the class and I started. And the other one I was just meeting out there, and we were at an event where I was teaching, and they said, "Look, really sorry, we try to pull everybody together but they're always late," and all that.

 

0:02:31.4 AS: And I was just like, this is interesting, the difference here. 'Cause it's the same cohort of people, it's the same group of executives and smart people in Thailand that are pursuing a degree. And the guy asked me, and I told him the story about the other university, and he said, "How do they do it?" And I said, "Well, they set a pretty clear standard of, look, this is important to us that you're on time, and we're gonna lock the door, and if you're not there at the time that it starts, you can't go in until the break. And we're gonna get class leaders to support this, we're gonna get alumni to support this, to say, this is part of what makes us unique."

 

0:03:06.9 DL: There you go.

 

0:03:08.1 AS: And I saw a very different outcome.

 

0:03:10.7 DL: So there you go. That's an operational definition. And whether or not you agree with it or not, you can see by having that operational definition at the one university, you've got a level of function that you don't have at the other university, you got a level of dysfunction, because they haven't taken the time to really do that. Really define what does that mean? And so when that happens, then you're dealing with all this variability, variation from students, variation from professors, variation from everybody in the system, and the overall system is not optimized. So it just keeps coming back to what we talked about before, but in this session we wanted to get into a little bit about, how would you begin setting an operational definition? What does it look like?

 

0:04:06.1 DL: When I work with, say, elementary teachers, I say, start the very first day, the very first thing, so... And start operationally defining what kinds of standards and what kinds of things you wanna have happen. Something so simple about, "Every time you hand a paper in, we want you to put your name on the paper." Okay, well, let's have a discussion about, let's operationally define, what does it mean to put your name on a paper? And sometimes people look at you and say, "Hey, what are you talking about?" Well, do you want first and last name? You just want a first name? Do you want it in the upper left-hand corner? The upper right-hand corner? Do you want it just anyplace on the paper, you don't really care? Well, if it's just random, if it's just anyplace on the paper, but my name's on it, well, that means that you as the teacher, every time you get a paper you're gonna be searching, trying to find somebody's name, right?

 

0:04:58.9 AS: Does it need to be clearly written?

 

0:05:01.3 DL: Yeah. Yeah, and what is clearly? I have no idea what that means. So we might have to have a discussion as a class and start to talk about, what does clearly written mean? There are some things to that. Well, clearly written means all the lower case letters need to be the same height. Oh, okay. So I'm thinking about that and I say, "Is my name clear, all my lower case letters the same height? Well, no, they're not." Okay, well then I can fix that, can't I? Right? So you can get clarity around these things if you really are thinking like this from the very beginning of bringing people into a system. As things get more and more complicated, and let's say that you wanna have a whole group of people come to a common definition, or a common operational definition on something. I started a process years ago of, in my classes, working with students and staff too, but anytime we needed to define something everybody would write it down, and then we'd pass it around and start to share those definitions and begin to talk about it.

 

0:06:20.9 DL: But then over the years, that sort of evolved into a tool that we call... Nowadays we call P³T. P to the third power T. So what it actually stands for is, the P³ is the paper passing purpose tool. And that came from one of my students one time, he said, "Oh, this is P to the third power." And so we just named it that, and it's become a popular way to define things. But what you do with that... I've done this with school boards. They wanna define certain terms with the school board and everything. So let's do a P³T. So what you do is you take... You wanna get everybody's opinion without it being tainted by other people, you just start having a big discussion about how you should define something or you can take a vague word, like we talked about last time discovery, but you could take a word like behavior or discipline or anything you want and you realize that everybody's got their own definition of what that means, but to optimize the system, we all have to have a common definition of what that means.

 

0:07:33.6 DL: So you take all the people in a group, usually try to... If I've got more than about five or seven people in a group, then we'll break that up into multiple groups, so it might have... So let's say I'm gonna do this in a classroom, and I wanna get everybody in the classroom help contribute to an operational definition. We might just put everybody into groups of five to say eight people, and just start the process, and so then we state, "Okay, well, we wanna have a term on a quality work." There's a vague term, we want all the work...

 

0:08:14.1 AS: Good quality around here.

 

0:08:15.7 DL: Yeah, quality work, we want everything that comes in to be quality. So well, we better define what that is. So first thing I'm gonna have everybody do is write down their own personal definition of quality work, right, and they're gonna write it on this paper, and then very simply, everybody then just passes it to the right or the left, and when you get somebody's paper, then the first thing you're gonna do is read that person's definition of that term. Whatever it is you're trying to define, and as you're reading it, you're gonna automatically come to certain phrases or words where you have an affinity for that and you're gonna go, "Oh yeah, yeah, yeah, I agree with that." So if you agree with that, what you do is you just underline it, okay, and so when you get finished, that paper has a bunch of things underlined and some things not underlined and then when you're finished with that, you pass it to the right and everybody keeps passing your paper around until you get your paper back.

 

0:09:14.5 DL: And if something's already been underlined once, then when I get it, if I also agree with that then I'm gonna underline it again, or if there's not enough space for that, maybe I'll put a check mark by it or just something that identifies that, okay, I agree with that, or I think that's a good idea, right? So when I get my paper back, I actually have the feedback from five to seven other people about what they thought about my definition, which is pretty interesting, right? Did I have a lot of people agree with me on certain things or maybe things that I thought were really important, nobody agreed with me on that, or nobody underlined that. They didn't think that was important at all. So you got all these definitions and papers, but you've got a chance for people to see everybody else's definition. And now we wanna start combining that into an operational definition agreement with the group. If you try to...

 

0:10:16.4 AS: And before we get to that.

 

0:10:19.0 DL: Go ahead.

 

0:10:19.7 AS: Before we get to that step. Can I just highlight? You talked about this idea of each person writing down their opinion so that they're not tainted by the influence of others, and that really is such a powerful first step. Because you could imagine that somebody would write down, "We've gotta be on time. That's what makes quality work to me." And now you end up... You start to expose pet peeves, maybe you call them, and then once you start... That starts going around those pet peeves kind of all of a sudden you realize that nobody underlined that, and then you think, "Oh, okay, that's interesting, I guess that's really important to me, but maybe it isn't." So that first part of the process that you've just defined of passing that paper around, I think is really valuable. So now, let's imagine that that paper has now come back to you. You're going, "Holy crap. I see some things that people agree with, but nobody underlined some of the things that I thought was important." Now you go to that next step. Tell us about that.

 

0:11:21.5 DL: Yeah, well often when I start with groups, a lot of times they'll have somebody that says, "Well, can we just discuss this before we start?" No, no. Absolutely not, because I don't want to give you the chance to intimidate somebody else in the group. Well, everybody here just knows what on-time performance is, right? And everybody here knows that it's this, and everybody here agrees with me. No, so psychologically, you don't wanna give people that chance to dominate a group ahead of time, and a lot of times our meetings are like that. Just full of that stuff. And by the time you actually get around to doing something or doing some work, people are so jaded, so upset that they just don't say anything, they don't wanna participate.

 

0:12:14.3 AS: And that's because let's say one person is kind of... They're persuasive, they have strong opinions, they tend to dominate of what it is, and we can say... There's a common word I hear a lot in America is inclusive, and one of the objectives of those people maybe is not to be too inclusive, and what you're saying is that when you're passing around that paper, it's a raw experience, you are forced to be inclusive of all people's ideas. And then you start to...

 

0:12:44.6 DL: And opinions and you may even have it, this is common students, that kids blow off things right, they just... They don't do it, or they put down something frivolous or silly or whatever it might be. Well, when they get their paper back and nothing on there was underlined, they start to get the picture like, Oh, okay, well, that was kind of a waste of time, right? That was just silly. And so next time around, you're probably gonna see that person take things a little more serious, a little more focus, think about things themselves before they get started, and you haven't had to say anything. They got the picture. Alright, so you got this paper back and you got these certain things, so now how do we move from... So we started with the individual and then we went to the group, and then how can you move that to a whole, say a whole class or a whole system depending on the size of the group.

 

0:13:39.9 DL: So probably the simplest way I've ever done it is I'll just ask somebody to start and say, "Tell me one thing that is underlined on your paper." People say, "Underlined a lot or a little or?" It doesn't matter. You decide, what was underlined on your paper that you think is important? And they'll tell me something, so we'll write that on the flip chart or the board or whatever it might be. Then you go to the next person and you say, "Okay, what do you have that's different?" And then you go to the next person and you say, "Okay, what do you have to add that's different?" And what you're doing is you're removing redundancies, you're removing all kinds of things as you go through, and people are starting to think harder and harder and look at their own paper and start to say, "Okay, these three things have already been listed, so I guess I don't have anything else to add." So once I go all the way around a group like that and I've removed all the redundancies, now we've distilled this down into what this group thinks is really important.

 

0:14:50.4 AS: And when we list those down, we're now listing down everything that's different. So we could have seven different things on that list.

 

0:15:00.0 DL: Oh, yeah.

 

0:15:00.2 AS: Would that be correct?

 

0:15:00.3 DL: Oh yeah.

 

0:15:01.1 AS: Okay. So now we got a long laundry list kind of thing. What do we do next?

 

0:15:06.7 DL: It can be, or it actually usually comes out much more concise than you think it's gonna be.

 

0:15:15.6 AS: Okay, because they've already kind of brought it down by underlining...

 

0:15:19.0 DL: That's right.

 

0:15:19.6 AS: And not underlining. Okay.

 

0:15:21.2 DL: That's right. There's other prioritization tools you could do and all kinds of things, but it takes a lot more time and etcetera. So now I've got this list.

 

0:15:29.3 AS: So let's say what, three... Two to five things that people have said?

 

0:15:34.0 DL: Yeah, it could be more like nine, 10, 12 things on this list.

 

0:15:38.7 AS: So we've got a long list?

 

0:15:40.2 DL: Yeah. So now how to... Let's say that I've got six groups of people in one room, each group has 5-7 people, but I wanna end up with a common operational definition for the whole room, right? Then I simply go to one group and say, "Okay, what do you have on your list?" And basically, I'm doing the same process but with groups. And this group says, "Oh, well, we said this in our group." Okay, so we're gonna write that down. I go to the next group, "What do you guys have that's different?" Then I go to the next group, "What do you have that's different?" And you just keep going around until everybody's... There's nothing left. Nobody has anything left.

 

0:16:18.6 DL: You talked a little bit about standards-based learning, how ambiguous that is. Well, here's a great way to take a whole staff and say, "Okay, well, let's try to define what we think this is, what does standards-based learning mean?" So first, what does it mean to you as an individual? And then how does my opinion sort of... What's the juxtaposition of my opinion with a group and then the thinking overall with the bigger group? So if I do the exact same process now with a group of 30 people within 10-12, 15 minutes, we've distilled it down to all the kinds of things that we think are really super important of this concept.

 

0:17:04.6 AS: Can I just go back? Okay, so first thing we did, we had... We broke people into different groups. So let's say we have, I don't know, five groups of six. And each person wrote down what they said their definition of that particular thing is, quality work or whatever, and then we push it around in a circle and people underline the things that they agree with, then we have... We then go up to the board and we say, "Okay, let's just get a laundry list of the different things," and we may have five, we may have 10, we may have 12, whatever, we have our list. Now, all of a sudden, you got other groups that have their lists. Now, what I'm trying to understand is that the next step, when you bring the other groups lists of 5-10 different things, are you...

 

0:17:49.4 DL: So basically, I'm doing this exact same process. I'll go to one group and say, "Okay, tell me something that you have on your group list... "

 

0:17:56.8 AS: That's different from this.

 

0:17:58.0 DL: "That you thought was important." And they're gonna tell me one concept. And then I'm gonna go to the next group and I say, "Okay, what do you have that's different?" And they'll tell me something else. Then I go to the next group, "What do you have that's different?" So I'm doing the exact same process, only with groups. And so I'm limiting redundancies again.

 

0:18:14.6 AS: And let me ask you a question. If you ended up... When you did the first group, you ended up with let's say 10 things. Then you go out to a group of groups, then you're gonna add probably on another one or two because one group had something that was different. So now you've got a list of 12 things or what happens by the end of that process?

 

0:18:35.0 DL: Could be. It's gonna distill it down, and sometimes it's phrases or even sentences or things that you wanna have in this operational definition. So at the end of this, you've got this concise document. Now, it's not a flowery paragraph or a statement or anything like that yet, but it is a list of everything that everybody in this room said is very important with this operational definition. And everybody in the room should be able to see how my contribution was folded into the whole. Because I can look at that list and go, "Oh yeah, I was the one. I said that."

 

0:19:15.8 AS: Because we didn't eliminate... Well, we could have eliminated something. If you had an idea, for instance, about quality work, let's say that it's on time and nobody else underlined that, well, then you can say, "Well, the group really doesn't see that as valuable as I saw it. So, okay, that's off." So we've eliminated some maybe pet peeves or frivolous things, and now we've a solid list.

 

0:19:37.1 DL: Yeah, and that's exactly what happens.

 

0:19:38.2 AS: We've got a solid list.

 

0:19:38.5 DL: Yep, that's exactly what happens.

 

0:19:41.4 AS: And then once we've got that solid list is what we wanna do then is just say, this is a list of everything, or do we then prioritize it, reduce it down, tighten it? What do we do from that point?

 

0:19:53.2 DL: Yeah, the answer to those questions is, it depends. [chuckle] So it really depends on what it is you're trying to do at that point. I find that a lot of times we're trying to operationally define something that's been kind of vague in the past, once we get that list and get it to that point, that's about all we need to do right now. We could spend weeks or months arguing about commas and coming up with some kind of statement that brings it all together, but you don't really need to do that because we've all agreed that these are all the things that we think are really important about this concept, and so that is our operational definition. Now, when I used to do this with students in classrooms... Yeah. But I'm gonna define something like quality work or on-time performance or any of those kinds of things or tardiness or anything. I'm trying to get rid of problems, right?

 

0:20:52.0 AS: Right.

 

0:20:52.7 DL: And so I'm gonna do any of those kinds of things. At some point, I would stop and have everybody take that list and try to combine those things into a paragraph or a statement.

 

0:21:03.3 AS: Okay.

 

0:21:04.8 DL: And have great fun doing that, where everybody's creativity and taking that list of items and turning it into some kinda paragraph that incorporates it. And students are amazing at this stuff. They come up with the most interesting concepts and ways of phrasing things that you could put a group of adults in a room for a year and they'd never come up with something so interesting. So, usually, at that point, I just have people go around and read what they wrote, and maybe we'll take certain phrases, and we could. We could turn it into a paragraph or something that was... Combining all that. But we're using that as a learning experience about how to take concepts and create definitions and paragraphs out of it.

 

0:21:50.6 AS: So let me try to summarize what we talked about. First, you highlighted the idea of like, wait a minute, what system are we trying to improve? We need to understand that first. And then to optimize that particular system, we need some common definitions. Now, when we started the conversation, I thought we were gonna end up with some really narrow, tight definition. I kinda was interested about where you ended. But before we get to where you ended with this, you talked about doing what you call P³T, or one of your students called it. I think you said paper passing purpose tool.

 

0:22:24.8 DL: Yeah. P to the third power.

 

0:22:26.1 AS: Yes. P to the third power T. And basically what you said is, "Don't let that start off as a discussion, because maybe one person could dominate that or try to influence what other people think about it." Rather, get each person to write down their definition of, for instance, we used the idea of what is quality work. And that's a pretty vague thing, so that's a good one for people to write down what are their opinions on it. And that way, they're not tainted by the influence of others. Once they've written it down, then pass it to their right, and let the person on your right underline the items on there that they agree with, and then pass that around. And by the time it comes back to you, you'll find that some of the things that you highlighted are agreed with, and some may not be. Then you basically take that and you go up on the board and say, "Alright, let's start with you." Start with one person and say, "What's one thing on your card that you've written down," and you wrote that down, and then you go to the next person, say, "What's one new thing on there that was not... That's not that?"

 

0:23:26.3 DL: Something different. That's different.

 

0:23:28.8 AS: Something that's different. And then you come up with a little bit of a laundry list. It could be five, it could be 10, it could be 12 of different things from that particular group of people. And then you can take it out to a bigger group where you have a series of groups that are doing the same thing. You then go around, you may add some things onto it. And then where I thought it was interesting where you ended with this, David, you said... You were kinda like, "Sometimes you don't have to go further than that right now. Just that is valuable process." And I thought, yeah, that's interesting, because just doing that, you can say we never have to do... 'Cause remember, before I was talking about discovery, and you were like, "I don't know what discovery is. I don't know what you mean." Well, you just described a pretty good process of discovery of what everybody thinks. Now, we don't ever go through that again. Do we have to tightly define that beyond that right now? Maybe not. Maybe we revisit it six months from now and tighten it up.

 

0:24:24.1 DL: It depends. So let's put this into practice. This is gonna be our definition of quality work. And so now we're gonna put it into practice, and then maybe later on, we can come back and re-look at the list. I've found that when you just have things in a list like that, people are more apt to wanna change it later on than if you have this flowery, nice paragraph that somebody's really worked on everything, and then people are like, I don't really wanna do that. But a lot of times, later on, after it's in practice, people come back and say, "Oh, we said this, but really, that's not really even relevant anymore." We can actually cross that thing off the list. That's not the most important thing. Or you were asking about prioritization. You could do a follow-on, you could do an NGT prioritization, nominal group technique, or you could do... Use just sticky dots to prioritize. Or if you need to. If you need to. But sometimes, most of the learning is in the process. So since every single person was a part of the process, every single person at the end of the process knows the definition of something, because they were a part of it.

 

0:25:35.9 AS: Yeah, yeah. Well, for the listeners and the viewers out there, what a great step-by-step guide that we can all try to put into practice. But most importantly, it kinda took the intimidation of operational definitions, it took some of that away from me that we're not... It doesn't have to be some... We've worked for hours crafting this statement. No. Here's a list of what we think is important. And that's good enough for a first step. And that, ultimately, is what we're talking about in this particular...

 

0:26:09.1 DL: And it's fun. It's fun. [chuckle]

 

0:26:11.8 AS: Yeah. And it's inclusive. It's inclusive. Well, David, on behalf of everyone at the Deming Institute, I wanna thank you again for the discussion. For listeners, remember to go to deming.org to continue your journey. Listeners can also learn more about David at langfordlearning.com. This is your host, Andrew Stotz. And I will leave you with one of my favorite quotes from Dr. Deming. "People are entitled to joy in work."