A Rigorous Alternative To Tests

How do we measure not how smart someone is, but how they are smart? That's the question that Elliott Washor and Big Picture Learning have sought to answer with their innovation—the International Big Picture Learning Credential. Exactly how they have built this set of credentials and made it valid and reliable is the topic of my conversation with Elliott—and what it might portend for the future of learning. As always, you can listen to the conversation here or on your favorite podcast player (just search the Future of Education), watch the conversation at YouTube below, or read the conversation below in full if you’re a paying subscriber.

Michael Horn:                Welcome, welcome, welcome to the Future of Education where we are obsessed with helping all individuals build their passions and fulfill their human potential. And today's guest has been doing that for a heck of a lot longer than I've been involved in trying to transform education and he's been doing the work in a variety of ways, really by pushing the envelope or even ripping up that envelope and throwing out the playbook altogether and innovating. His name is Elliot Washor and among the many hats that he has worn and had, he's the co-founder of Big Picture Learning. He’s the co-founder of the Met Center in Providence, Rhode Island. And we of course have had learners from Big Picture Learning schools on the show we've had Andrew Frishman on the show. But the topic that I really want to dive in with Elliot on is the international Big Picture Learning credential, which in the words of Elliot to me, “Changes the equation by measuring how an individual is smart, not just how smart they are.” And I think that's a really profound flip and an exciting set of opportunities. So first, Elliot, welcome. It's really good to see you. I assume you're coming to us today from San Diego?

Elliott Washor:           Yeah, it's not what people think. It's raining and chilly here today. I got a sweater on even. I don't mean to make anybody feel bad who's in real cold weather, but not all is perfect in San Diego like people think.

Horn:                Well, still close enough to paradise I suspect. But before we really dive into the nuts and bolts here, let's talk about this credential that you all have created. It was developed by Big Picture Schools, as I understand it in Australia, and then the University of Melbourne I think partnered on this to sort of vet and bless it if I understand. But tell us a little bit more about what it really is and why it's unique.

Washor:           And I feel that over the five decades that I've been working that... And we've all in very good ways, many of us, struggle with assessment, constantly that our colleagues in Australia and the Viv White who's the co-executive director of Big Picture Learning in Australia and her staff and we have about 40 schools there, have come up with something that contribute a lot to the field. So what happened was, is that Viv took Big Picture's learning goals, which are basically pretty easy to understand by anybody in that we like to credit students around how they apply their academics in and outside of school with mentors and teaching staff as advisors in the real world and in school, once again in communities.

                        What they did was they took the learning goals and they went to the University of Melbourne to the psychometricians there, the head of whom is Sandra Milligan. And they said, "We want to credit teacher judgment, student self-assessment and mentor judgment and we need a warranting and validation entity to do that." And Sandra and her team collaborated with our schools and developed an algorithm that as you put the information in, through once again teacher judgment, student self-assessment and mentor judgment, it spun out a validation that vetted and warranted that this was real, that what they actually said they did, they did. And then our students applied to universities, colleges, technical schools, and the workplace in Australia and it lo and behold it was accepted without using standardized test scores or GPAs.

Horn:                These areas now that they're vetting are things like communication skills, empirical reasoning, quantitative reasoning as I understand it, personal qualities and sort of metacognition or knowing how to learn, I think. What are those artifacts that they're feeding in or into that algorithm really look like?

Washor:           So basically it is students create their own learner profile, which is both text and video, that's a piece of this, and that can start and we see this starting kindergarten all the way through. Right now we're doing some work at the K to eight level with this, excuse me, but it's mostly at the high school level. And when a student has compiled their work, both they're in school work and they're out of school work, which for us are connected, they put it together and run it through what they call this pedal or ruby, which looks like those six measures in them. The learning how to learn, one is metacognitive as well as disciplinary thinking, how to learn to think in a discipline.

                        And once again, all that work is looked at and vetted all the way through. And then in the junior year and senior year it goes into the University of Melbourne to be vetted and warranted and they somehow spin it out with an algorithm that they're working on to get better and better. And it's pretty good now. And lo and behold, it worked. It gave enough validation and credibility to what many of us needed.

                        So for example, if you have a portfolio from any school, how does that university, workplace, technical school know that it's real? And that's the function of what Uni Melbourne is doing. And I don't know why we never thought of it like that before. It seems pretty simple and smart what Viv and her colleagues came up with, but we hadn't. And so we just relied on sending out transcripts if you will, and portfolios. Now if there really is, and there is developing a clearing house around this, we can utilize a university, a trusted place, to vet and warrant these three things around student learning. So now what happened was in the research and at the end we can point you to the research that's been done in Australia by third parties, we found out that indigenous students, immigrant students are getting into places that they normally wouldn't get into, because it's not about how smart you are, how you look to a test, but it's how you are smart. And all of a sudden people are starting to recognize that and go, "Wow, this is really, really different. This makes a lot of sense."

Horn:                Super interesting. And as I understand it, in each of those six areas, there's five levels. So I can say in effect, "Here's my portfolio of work, the things I know and can do and show application of." And it's going to say, "Great Michael, you're a level four communicator." I'm making this up right. "You're a level three in your quantitative reasoning." Whatever it is. Basically these mastery thresholds.

Washor:           Yes.

Horn:                I assume. And then that's going to create a very simple portfolio of who I am. I say simple in the sense of it's a quick representation for a university or employer or whatever. And then they can dig into it to start to say, "Okay, what does that mean Michael can actually do and let's actually look at the work in itself and the exam."

Washor:           Yes. And before it gets to that point, there's professional development where staff from the same school and staff from different school look at different students work and rate it. So there's that piece going into that algorithm as well. And then that university psychometric work is saying, "Yes, it is what they say it is." And that's really powerful and important. So there's a lot to it that makes teacher professional development and judgment, it raises the bar on that, if you will.

                        And what we're finding out, and we've started it in the United States with about 25, 30 schools in California, Washington state and New York, is that the algorithm, the teachers are harder on the students and the students are harder on themselves than the algorithm is. So there's all this play going on, which I feel is really very healthy.

                        Now, the implications for that are pretty amazing, because now you can credit learning that happens outside of school, as well as inside. Now you're crediting tacit learning, where you know more than you can say, you can show things. Now you're crediting fluid intelligence and young people recognizing patterns that come out culturally and in their communities and from their neighborhoods and families. So it looks, it's taking this to another level. We've also instituted it in or started it in where we have schools in Barbados and Kenya and that it's working very well there as well.

                        So what we're looking for now and what the University of Melbourne is doing, Sandra Milligan and Viv is we're looking for university partners in the United States. And Uni Melbourne has sister relationships with other universities in the states and that's where Andre's going first. And we're seeing how they can partner with universities here. So the transfer of student data is not going across internationally. And of course there's all these pieces that have to be worked out around online student data and all that and privacy. And that's also a part of this work as well. And it's not easy work. It's rewarding, it's exciting, I hope it leads once again to other people figuring this out even better than we figured it out. But we're going to keep pushing on it because we think that Viv and our colleagues were really onto something.

Horn:                Well, so there's a couple pieces there that I want to come back to, but just, you talked about what it starts to replace a little bit. Standardized test scores that are very narrow measures of certain basket of skills, we'll call them, grades that maybe are unreliable because maybe the teachers are underrating what the students can do or vice versa in certain settings, I'm sure and grade inflation. But that it's creating these robust measures. It strikes me that something else that it's doing is that this isn't just say badging work that's been out there for years because there's this validity and reliability behind it. It strikes me, we see industry certifications, but what seems different here is that these are... The six areas you just talked about, those are sort of school competencies if you will, that should port across states, countries, school systems, et cetera. And then there's stuff like the Mastery Transcript Consortium and stuff, which I think is super exciting, but they're not actually doing the validating, if you will, of the actual learning itself as far as I understand it. So it seems to maybe not replace but fit within something like that. Do I have that right? What else is it replacing and tearing it out?

Washor:           We've been having some very nice conversations with Mastery Transcript and other platforms like Capable and it does fit because we're adding some value that quite frankly people haven't thought about in a way we have. And what's one of the things that's most interesting personally for me and for our work at Big Picture is that crediting outside of school learning that it doesn't just happen inside the four walls of the school. That certifications are out there in the real world as well as in school. And that becomes part of the process of bringing this work back in school and crediting it and accrediting who the young person really is from the community that they come from. All those pieces are in incredibly important that are leveling a playing field.

Horn:                Well, and correct me if I'm wrong here, I have this recollection that used to work in New Hampshire even before Big Picture Learning days. If I'm correct, I don't know where I'm pulling that from. But a place like New Hampshire now has these pathways to credential out of school learning. I think one of the reasons maybe people have had concerns about it is that there's not been an assessment sitting on top of it to say like, "Is this really just Michael Horn's volunteer hours at a community center? Did he actually know and do and apply something?" And it seems like this framework could also really help support those more systematic efforts as opposed to say, individual schools or school networks that have valued out of school learning. This could really turbocharge some of those efforts to be a little bit more true I guess, to this notion that we all know is true, which is learning happens all the time anywhere. We just haven't figured out a way to credential it, if you will, outside of the formal school day.

Washor:           Yes, to what you said, I feel that whether people buy into the work that is the international Big Picture Learning credential or learn from it, it's once again adding value to the whole body of work that's out there and it's needed and necessary if we're going to move this forward. I always say on the competency based systems, they talk about anywhere, anytime, but if they don't talk about the many ways that you really understand something that sometimes you can show it but you can't say it, which is how you work more in the real world, then we are bouncing back to a written text test or/and text for any kind of validation and certification.

                        And this is way more than that. You can actually show what you can do. As Wynton Marsalis said at one time, "There's only one school and that's the school of can you play?" And that you can't schools shy away from that. It has to be what you put down in a written test and what you say. But so much of the world works in the tacit arena, in the arena of a craft, if you will, and a science and an art all integrated together. Whereas our measures are text-based and written and spoken, but not necessarily something that comes out of a community of practice outside of school that is validating, "Can you do it? Is it good enough?" And those pieces around is the, my buddy Mike Roses who passed away would say, "As mastery as foregrounded, the clock recedes." When you looking in school time and a set time of a day and a set period and a set number of weeks and then you're punching out a grade that looks very, very different now, than something like the IBPLC.

Horn:                So I want to dig then into the valid and reliable part of this. And I mean a psychometrically evaluated tool, it seems to me that that's pretty breakthrough here. It's another thing that's different from all the badges and stuff that's out there because it says, "Yeah, the student can do these things." And you mentioned interrater reliability of having different places look at an artifact of work and saying what this really is a mastery threshold of and so forth. Tell us more about, how does someone know if it's valid and reliable, what does that really mean and indicate because we hear it all the time as sort of the gold standard of assessment.

Washor:           Yeah, no, it's a really good question and I'm not a psychometrician, but I'm glad that some psychometricians are interested in exploring this with us. And I feel that they have really put their heart and soul and analysis, analytics to this, to really come up with something that is ongoing and developing, but is pretty damn good. And so using what is being inputted by teachers, by mentors, by students and creating that analysis is what's doing this and looking at bigger numbers of students going in to this process is what's going to make it more, is going to refine it and make it even more reliable. I can say for sure, pretty much for sure that much. Like I said, I know it's not magic, but I know, and I know that there's a really serious bunch of people playing and involved in producing these kinds of analytics.

                        So I think they're on their way, it's real important. And for some reason we'd all like to say, "Just use teacher judgment." But it hasn't happened in 70 years and it ain't going to happen. So we come up with something called standardized tests, which measures things way, way too narrowly for most of us. It miss too much. This is something that's giving us this other kind of breakthrough, possibly, disruption, possibly. I don't think we're at the end of it, I think we're at the beginning of it. And I really feel very hopeful about it and I hope, and my hope is that we get the supports and that people, because people say, "Hmm, this is really important." I can't keep doing what I'm doing like you discussed in New Hampshire, it's not going to give me what we really want. It's not going to give it across states, countries, even within our own place.

                        And this is that whole piece around, as well, around our work and that's been in the field and the research work of people like Julia Freeland Fisher, it's who knows you know what you know, and that gets you to a place, well that's that outside of school learning validation that's coming into this, as well as you hear the student voice in the learner profile in the video and in their written work of them telling everybody who they are and who they want to become and what adds meaning to their life. And it's different for each and every young person.

Horn:                Well I love that you can represent that individuality and that pursuit of uniqueness as opposed to trying to all compete to be the best on something narrow, which creates the zero sum mess that we're in right now. The question I guess I want to end on is, folks who listen to me know that I think innovation in an assessment is perhaps the most important or in the top two certainly innovations we need to enable a real positive sum system to flourish where we move beyond the timed measures and so forth and allow each person to be the most unique person they can be, ultimately. So you all have developed the beginning of this, as you said, you're bringing it into the US now you're looking for the university partner. What are the steps ahead? Are all Big Picture Learning Schools going to start using this credential? Or what's it going to take I guess to start to get this spread? Give us a little bit of the roadmap mapping.

Washor:           Right. So right now we have about 30 schools in the US and around the world aside from Australia involved. And although we use... Big Picture Learning goals are interesting in the sense that I think you were saying, and I would agree, they're not so abstract like a lot of competencies, like creativity. All right. I mean when does that happen? I go, ah, when I hear that, I go, it's wonderful, but these are really tied to the school. Empirical reasoning is applied science. So you applying your knowledge and use of science, use of quantitative reasoning, mathematics.

                        When we talk about communication, we're talking about not just written, it's like the Isadora Duncan line, "If I could say it, I wouldn't have to dance." So that's part of communication too. All those ways that we can communicate. And so we are closely aligned to what schools can grab onto. And now we're saying, "Ah, wait a second, now we're including application." And the application can happen inside or outside of school and that's different. That's key. And yes, we're going to use the judgment of mentors who talk to students and teachers and yes, student voice is going to be part of this as well. So yes, we're looking for, and Uni Melbourne is looking for a university partner. We're looking for platforms that we could put this on, that are next steps so it is not... It's not two steps over the chasm to get to applied academics, which make it a lot easier set of learning goals, if you will, that can be understood by any school system, public, private, home.

                        And I think that there's a big market for this in homeschool. I think there's a big market for this in youth development and in public schools of course. So that's a little bit of an answer to your question, I think, of places we have to go and how it can connect beyond Big Picture Schools as well as using ours as a pilot as well as we have schools in, and another part of Big Picture called Upstream Collaborative, which are not necessarily Big Picture Schools in full, but are schools that want to learn more about how we do what we do. And they're also involved in this as well.

Horn:                Love so much of this and I can't wait to follow the progress and have you back on at some point, I think, to update us on that. But for now, as we wrap up, where can folks learn more about the international Big Picture Learning?

Washor:           Go to our website, Big Picture Learning and look up the IBPLC and my email's, ewashor@gmail.com or send me an email and I'll send you, if it's not up there yet, I'll send you the research. You can go to Big Picture Australia, as well. They have a lot more posted than we do because it's their research. So yeah, there's a bunch of places you can find out more information. We're happy to talk to any and everybody about this and hopefully once again, like I said, if it's not us and somebody else takes this and goes further with it, in better ways, that's fantastic. But I really feel, once again, I give a lot of credit to Viv or colleagues that they really have contributed. And I'll say it might be a nice idea to have Sandra and Viv on to talk about those psychometrics because I'm fully admitting that I'm not the person to talk to about the details of how they analyze what they do, but I've heard them talk about it. I'm just not the person to talk about.

Horn:                Let's do that soon, we'll say. But for now, Elliot, huge thank you. Appreciate what y'all are doing and appreciate you and for everyone tuning in, we'll be back next time on the Future of Education.

Thank you for subscribing. Leave a comment or share this episode.