Redefining Metrics in Design

Transcript

Jesse: Hey everybody, it’s Jesse James Garrett here. I wanted to let you know before we get into the show, we’ve got a special event coming up this week on Friday, March 28th, we are celebrating the 25th anniversary of the elements of user experience with a live 90 minute virtual seminar at 8:00 AM Pacific, 11:00 AM Eastern.

I will be talking about the elements of UX in the age of AI. We’ll be looking at the connections between the history of user experience design, and the future of artificial intelligence. You do not wanna miss it. Peter Merholz will be there conducting live q and a. So please join us on Friday, March 28th.

Get your tickets now at jessejamesgarrett.com. Stay tuned for another special announcement at the end of this show, but now on with the show.

I’m Jesse James Garrett,

Peter: and I’m Peter Merholz.

Jesse: And we’re finding our way,

Peter: Navigating the opportunities

Jesse: and challenges

Peter: of design and design leadership,

Jesse: Welcome to the next phase. Joining us today to talk about what’s next for design is Dr. Sara Beckman, Professor at Berkeley’s Haas School of Business and longtime observer and commentator of the dynamic between design and business. We’ll be talking more about the legacy and impact, for better or worse, of design thinking, how design leaders should talk about metrics and how they shouldn’t, and what she’s learning from educating the next generation of product managers.

Peter: Hi, Sara. Thank you so much for joining us.

Sara: It’s a pleasure to be here, Peter.

Peter: To start off, while I’ve known you for over 20 years, our audience doesn’t. And so I’m curious how you introduce yourself these days.

Sara: It’s always a good question. I’m on the faculty at the business school at UC Berkeley, where I have, for multiple decades now, been teaching topics in design, innovation, product development, product management et cetera. I kind of hang out between two worlds there, between the College of Engineering and the business school.

I was involved in starting up the Jacobs Institute of Design Innovation. So a lot of focus, I guess, broadly speaking, on cross-disciplinary work in the university, particularly as it relates to design. The ability to create new stuff, I guess.

The Intersection of Business and Design

Peter: Excellent. I’m going to dive right in to something I was thinking about literally yesterday, where I was attending a session, it was a webinar given by a design leader talking about the intersection, wait for it, of business and design…

Jesse: ooh,

Peter: And how it’s important for designers to understand, and to be able to speak in terms of metrics and stuff like that.

And as he was talking, I was reflecting on work we did together over 20 years ago. So for Adaptive Path, you were kind of an advisor when we did a report on, at the time we called it, like, the working title was The ROI of UX, the official publication title is Leveraging Business Value: How ROI Changes User Experience. And the question I have is, why is it 20 years later we’re having the same conversation about how you connect design and business? Why doesn’t it feel like it has progressed? ,

Sara: Well, can I say something maybe a little provocative and say,

Peter: Yes!

Sara: Design Thinking got in the way.

Jesse: mm-hmm

Sara: So, 20 years ago, we were talking about, I’m going to call it real design. So, whether it’s UX designers, industrial designers, graphic designers, there was work that they did that we were trying to put value on. We were trying to say, if I make this product more usable, I can sell more of it, for example. So how do I make a connection between the deep work that designers are trained to do in design school, and the outcomes I can achieve with a product or service in the marketplace?

Design thinking came along, and, in my opinion, it trivialized the work of real designers, I’ll call them, and we said everybody can do design and we had all these things going on that we turned design into, frankly, to some extent, soundbites.

Oh, go talk to a customer and then design something cool and new. Nothing wrong with going to go talk to a customer. We kind of skip over a lot in design thinking the idea of actually getting insights out of talking to real customers and then designing to those insights.

I’m put in mind of Barry Katz’ book on the history of design in Silicon Valley and how it evolved. First, started at Hewlett Packard, where I happened to work way back when, as really usability or user interface, right, design.

And then as we moved into wrapping services around things, as we moved into software being the core of the delivery of capability, we migrated what design did, but design thinking was a whole different thing.

And so was looking through your recent interview with Roger Martin and, thinking about what is design thinking relative to what design was about. Sometimes they call them little d and big D design. That implies one is bigger, better than the other. But we used to do that in manufacturing. It was big M manufacturing, which we thought of as manufacturing strategy and sort of the wrapper that went around manufacturing. And I think that was different than the actual execution of manufacturing processes.

Somehow de-linked with design thinking, sort of the big D design stuff, we de-linked it from the actual actions of real designers. And that’s part of why people like Lucy Kimball talk about designerly thinking, as opposed to design thinking, because design thinking was a broader mindset and it left behind some of the roll-up-your-sleeves, we have real work to do here, because it made it seem like I could just draw a journey map and then design something, or I could just brainstorm for a bit, diverge, converge, and come up with something. No design process is that easy, right? I mean, you both know this from…

Jesse: Right. Yeah.

Sara: And so we left all that behind. Well, what’s the ROI of design thinking? First of all, nobody even knew what it was, right. I mean, you look at the academic literature on design thinking. I always felt like this was a bit of a tautological. Oh, well, let me go study companies that say they do design thinking and then I’ll define design thinking and then sort of over time we defined this thing called design thinking, but then next thing you know, design thinking isn’t just design.

Oh, it’s also teaming. Oh, so design thinking is going to resolve all the teaming challenges. Like we, it just kept getting bigger and bigger without an anchor around what is this thing. It became bigger and undefined, but we also lost track of… design thinking, which is different than the practice of design, is not the only way to frame and solve a problem.

And Roger talked about, well, scientific method is another way to frame and solve problems. Critical thinking is another way to frame and solve problems. Total quality management or DMAIC or Six Sigma, whatever you wanted to package all that stuff in, that was another way to frame and solve problems. Systems thinking is another way to frame and solve problems.

And the design thinking proponents came along and said, this is the way to frame and solve problems. And it became, therefore, a fad, right? Because, because you can’t say there’s only one way to frame and solve problems. This is one of my great frustrations, honestly.

So that’s what led me to the model that I’ve been using almost 20 years now, which is this experiential learning based model that basically says problem framing and solving is a process of sense making, toggling between the concrete and the abstract world, observing and noticing, framing and reframing the problem, and solution making, agains toggling between concrete and abstract worlds.

Sara: I have a great idea in my head. I go try it out. I can put all of those other problem framing and solving methods into that model, which then led me to say, well, then shouldn’t we be teaching students, at a generic level, the core capabilities of problem framing and solving, and then have them say, ah, this problem could use scientific method approaches. It’s hypothesis-driven, right? I observe customers or users and I create a hypothesis about a problem they have. And then I go test that hypothesis by playing around with different solutions and prototypes. Scientists observe nature, maybe. We observe people in the design world, but they’re observing something and then forming a question about how it could be different.

So, Sal Khan, in his book, One World Schoolhouse, his last chapter says, What if the university were a place in Silicon Valley that companies threw problems into and kids solved them for four years? And it took me a while to really kind of digest that notion, but I think, and by the way, it doesn’t have to be in Silicon Valley, and it doesn’t have to be corporate problems, but a general notion that education would really be about iteratively framing and solving problems of different types of different magnitudes over time with different methodologies. I think of different mindsets, different skill sets, different tool sets, right, that students would then learn in this setting.

A systems thinking mindset, skill set, tool set is a better way to tackle this problem than in this other situation over here, where scientific method might be more applicable set of mindset, skill sets, tool sets.

Taking The Time to Get It Right

Peter: So you’re laying out a robust, detailed, savvy framework for approaching, tackling problems. What I’m wondering is, I don’t think I have ever been exposed to a company that would engage in the kind of rigor that you are proposing as they work through their challenges, and so, what’s the misfit there, right?

There’s a misfit, I mean, you’re a B school teacher, I know you do exec ed. So you’re talking to people, not just students who’ve not necessarily had real jobs before, but people who are in the working world who are trying to get better at it, you’re engaging with folks with real world challenges.

Square this, you know, thousand points of light, very robust framing of all the ways an organization can consider how to best tackle the problem in front of them.

One of the reasons I think design thinking became popular is, in part, because by underselling design, it could be done quickly. And these companies just want anything that can be done quickly. They don’t want to spend a lot of time thinking upfront. You might not be familiar with this. There’s this saying “big design upfront,” which is like a bad thing, right? You’re going to spend three months doing all this work and designing, and then we’re going to get to start building it.

So square the rigor and robustness that you’re discussing with what we witness every day with companies who are barely able to tie their shoes, much less engage in any meaningful research to inform how they solve problems.

Sara: Yeah. I mean, I think that’s that’s a fair judgment. I teach hundreds of product managers every year. So I watch a lot of this. Clearly time matters, but there’s a couple of things that I try to do here.

One is, you know, I used to teach design for manufacturability forever ago, my industrial engineer side of me used to teach operations management, and we taught a case study about a disk drive company. These are commercial disk drives, right? So they’re very, very high volume, low cost product, right? And this company was working in a partnership with Japanese company that was doing the manufacturing. And the engineers at the Japanese company would say, We need you to properly spec this product. And the engineers in the U. S. company would say, We don’t have time to properly spec this product. And the Japanese engineers ultimately won. They won over the ongoing complaints on the part of the U. S. product development team who they said they just want us to be able to put the parts on the table and hit the table and the parts will jump together.

Like that’s ridiculous. But in the end they spec’d this product and they were able to ramp to volume manufacturing in way less time, way faster ramp to very high quality levels, because they had done some amount of initial investment.

The parallel I draw to that in the design and product management world is that having some degree of clarity around what problem you want to solve for the customer or user up front will buy you a whole bunch of time in the long run, because you won’t keep putting the wrong thing into the market. Now, I’m not suggesting that I sit around and look at my belly button all day to do that. That’s where the innovation cycle comes in, right? I talk to a few customers. I take a step back. I go from dance floor to balcony. I get in the balcony. I say, what problem do I think the customer’s trying to solve here? Then I say, well, if that’s the problem, here’s three ways I could solve it. And I take those three ways back out, and I find out they don’t really have that problem. So I pivot, right?

Or alternatively, I could say, I observe three critical problems that the customer has. I’m going to then generate ideas for each of those three problems, take them out, and figure out the customer’s going to say, I want to take that one home. Then I know I’m solving a problem the customer cares about. So the whole notion of rapid cycle prototyping is super critical here. But to frame it in terms of, I’m not just rapid prototyping to create the best product, I’m rapid prototyping to make sure I’m solving a problem that someone cares about, and there’s huge value in that, right?

Jesse: What I hear from many of the design leaders in my leadership coaching practice is that their leadership is on board with many of the ideas that you’re describing. They are on board with being customer centered, with really listening for the right signals and tuning the product to the market in these ways.

They just don’t see design as necessarily a partner in making that happen. And I wonder about design’s value proposition as it relates to this value that you’re describing because, you know, as you mentioned, you teach product managers how to do product management. And there are many, many people who see all of these things that you’re describing as being fundamental to the product management role, much more than to the design role in the organization.

And so I wonder your thoughts on design, where design plays in these spaces.

Sara: So that’s where I have to go back to this question of so-called little d design, right? I mean, product managers are generally not experienced or not taught or trained in doing UX design, right? So who is their partner, to be able to say, put this kind of curve on the product and this will happen. This goes back then to Peter’s original question of, you know, back 20 years ago, we were trying to characterize those kinds of connections, not this meta design thinking thing, right. We could have a long talk about customer centricity and whether companies really are or whether it’s just a lot of words and right, so, there’s a whole problem in companies.

The Folly of Net Promoter Score

Sara: I use net promoter score as my sort of measure of that. It has nothing to do with customer satisfaction. In fact, there’s no correlation between customer satisfaction or increased profit and net promoter score for any number of reasons.

First, it’s ill constructed from a statistical point of view. It randomly eliminates 2 of the measures. It’s not even a proper average. I look at mine, I go, it got better and I think I’m good, but I don’t know that my competitors is 2 points higher than mine. In which case, getting better wasn’t really relevant.

Peter: I like how for you, net promoter score, what it indicates, isn’t customer satisfaction, but a company that is kind of foolishly trying to engage in the idea of customer satisfaction.

Sara: I mean, they think it’s an appropriate proxy, right? And I think that’s, well, it’s not an appropriate proxy for really understanding, are my customers now able to achieve an outcome better than they could before? Now, you might have a proxy for that outcome, right? But I want to have a way to know, have I actually helped my customer execute the jobs to be done that I intended them to execute.

As long as you have these broader metrics, it’s very hard to connect what a UX designer does, right? I mean, I think it was an Adaptive Path conference where I put a DuPont chart up there. This is very old stuff. DuPont charts. Accountants would know what a DuPont chart is. Like, It starts over here with profit, which is made up of revenue versus cost. Revenue is price versus volume. So it basically backs all the way through financial metrics. And I put that out there and I said, “So where do designers affect this?”

Jesse: Right.

Sara: It could be at any of those levels. Remember Sam Lucente at Hewlett Packard. He had these multiple layers when he went in as the head of design. He said, well, it was clear that I had to do cost reduction stuff to prove my value. So I standardized the logos on all the pieces of equipment that HP sold at the time. Saved a boatload of money. Also, by the way, created sort of a standard brand identity in the marketplace.

So he had, at one end, let me save cost, right, by good design, and the other end, he was working on revenue growth opportunities, right? If I can identify an interesting new need in the marketplace, then I’ll be able to grow revenue.

Connecting Design to Business Value

Sara: So to me, we have opportunities to make very explicit connections between what designers are doing. We had a discussion about the DuPont chart. And I was trying to get the designers in the room to think about the CMO, the chief marketing officer as their customer. And I said, what outcomes is the CMO trying to achieve and how are you going to connect the design you’ve made over here to helping them achieve those outcomes? And it was really hard to get them to do that. I can’t remember exactly the details now or I would tell the whole story, but, in effect the person said, yeah, but it’s Claudia Kotchka’s, you know, put a gold rim around the top of a face cream so I can put it in CVS or Walgreens and have it look more upscale, and everything, there. And the executive saying that cost two cents per jar, take it off. And her having to say, but then it won’t say that it’s…

Peter: Premium.

Sara: Premium, right?

Jesse: Right. So, in some ways what you’re talking about are really qualitative outcomes that design creates in this increasingly, as Roger pointed out, as has frequently been a theme on this show, in an increasingly metrics-driven world. And I wonder what you see as design’s relationship to business metrics.

Get Your Proxy Metrics Right

Sara: So when we teach product managers, we try to have conversations about this. There’s a great case study, and this is kind of the classic case of Kodak. What was Kodak selling? Memories, and they knew that. What was their proxy measure for you to capture and share memories? Two core jobs to be done. Capture and share memories, right? Proxy measure, how much film and paper do we sell? So what happened to Kodak? They got too hung up on film and paper sales as the proxy metric, and when we shifted to digital, they didn’t shift their internal proxy metrics.

They knew they were in the business of memories, right? They invented one of the first, if not the first, digital camera. they were on top of all these things, but when it came to execute, the metrics they had in place, they needed to be changed as a proxy for that.

So mostly we use proxies, right? Like monthly average users. Well, you know, monthly average users is just a number that customers don’t care about. What did they care about: learning from the site, or getting a job, or whatever it is. So, how do I get proxy metrics? I’ve always hated sort of measuring everything myself, like it frustrated me when I worked in business, so I’m not not here to say, I think you have to be able to measure everything.

But if you’re going to measure, then for heaven’s sakes, don’t use net promoter score, at least try to use something that gives you some information about the customers and whether they’re achieving something they want to achieve. And I think the Kodak story is a great example of knowing what you want to help, but not migrating or connecting metrics over time to the outcome.

Peter: So you say, don’t use net promoter score. I agree. Many UX types agree.

Jesse: I also agree.

Peter: A few weeks ago a head of design shared with me that their boss, in an effort to give design some accountability, “We’re going to have you held accountable for net promoter score.”

And this design leader was uncomfortable about that. I’m wondering what you would counsel someone, ’cause I can point to dozens of design leaders who are in a similar situation, what have you seen, if anything, that works to start changing that conversation? Shifting it, when you are also not the one in power, right? This is your boss saying this.

Jesse: Right.

Peter: You might be new. This person’s relatively new in the organization, right? Like, so they’re still earning trust, building credibility, like, there’s some things working against them to be able to just say, no, that’s not the right way to do it. It should be X.

Any thoughts on how you turn that ship, to start advocating for what you do believe is right?

Sara: Yeah. Many years ago when I worked at Hewlett Packard, HP at the time was divisionalized, had 50 or 60 manufacturing sites globally, which was a lot, and they were all very small, and so we were trying to provide some evidence that consolidation of those sites might be useful, so we made a cost volume curve. And where was HP? Above the curve, on the steep portion of the curve. So we start showing this to people and we say, hmm, opportunity, right? Increase volume, reduce costs. Oh, no, we do better quality work than that. Okay, how many quality engineers would you like me to add to the curve?

Jesse: Hmm.

Sara: And it would move up, you know, a teeny, teeny, tiny amount and we’d go at it again. For a year, we had these conversations. Until one of the very senior, senior guys said, wait a second, how many engineers do we have working in each of those sites?

I said, well, it’s like two per site. You’ve got a hundred across the company, but the two, whoa, if we put them all together, we’d have an awesome printed circuit board assembly site. Like, okay. So it took me a year to get there.

Two morals of that story for this conversation. You’re not going to change the boss’s mind overnight. But you keep bringing this up. So there’s a whole bunch, there’s articles–the wallet allocation rule. There’s a whole bunch of articles out there, first of all. So I would start with, there’s evidence of the problems of using net promoter score, right? So I can put some of those out there.

But probably more importantly, I would start trying to build up some understanding of how I can create a metric that connects more closely to what the customers and users are trying to achieve. So I would start thinking about, is it weekly active users in the example views, or is it weekly learning users? Ah, well then if it’s weekly learning, what do I actually watch people doing? Do they scroll more? Do they post more? What evidence would I have that they’re actually on the site learning something?

So I would start to build up a conversation around what can we replace this with. If you’ve got to have a metric other than revenue, right? If you’ve got to have a metric, then what metric might I have that would allow me to connect to something that is the agenda that I think I have for helping customers and users.

Jesse: And it takes time. So part of this is patience. I’m new here. I’m not going to walk into my boss’s office, although I’ve been known to do stuff like that, walk into my boss’s office and say, you know, this is dumb. You should think differently. But, you know, building up a case. Maybe being able to collect some of that data, maybe talking to the data scientists and saying, if I wanted to construct a metric that evaluates memories, whether they’re shared and whether they’re captured, do you have data that could help me do that? So why don’t the designers make friends with the data scientists and say, you know, if this is something I would like to be able to evaluate, do you have a way to help me do that evaluation? So we begin to get little better integration.

You talk about organizations basically falling in love with their proxies and becoming so attached to their quantitative measures that they forget what those quantitative measures were intended to stand for, which had to do with some sort of different kind of qualitative impact.

Jesse: I work with lot of design leaders who are basically trying to break that spell of having fallen in love with those specific quantitative proxies. The challenge with that is, and I love your suggestion that like, don’t try to convince them to become a qualitatively driven organization, just give them different proxies, better proxies makes a lot of sense.

The attachment that happens in these organizations is often connected to financial incentives. It’s often connected to power. And design leaders often find themselves literally the least powerful person in the room when having these conversations about how we measure the value of the work that we’re doing together.

And I wonder about your perspective on the arguments that are going to hold water in a situation where you are the least powerful person in the room, arguing for everybody else in the room to change the way they measure their success. How do you do that?

Find Your “Trojan Horse” Move

Sara: It’s hard. There’s no… it goes back to my HP story, like, I couldn’t change the whole company.

Jesse: Right.

Sara: I could only change it one person, one division at a time. And so we used to talk a lot about, what’s our Trojan horse move, what’s our subliminal messaging, like how do we create shifts in thinking over time.

And the kinds of things I use with product managers are, I can’t implement a big thing all at once. I have to often start small. So what if I open every meeting with a customer story? Oh, I was talking to a customer the other day. You know what they were doing with our product? What if every time somebody came to me with a feature they want me to build, I said, cool, happy to consider it, what’s the customer going to do with it? What problem are they going to solve with it?

So there’s a whole lot of really simple little things, and design thinking did contribute a whole lot of these, but I can change my own behavior, and then what happens, you start to hear other people starting their meetings with customer stories, or you start to hear other people saying, but what problem is this solving.

But we have, to your point, incredibly entrenched behaviors in organizations. And I’m going to take that all the way back to where we started this conversation, standardized testing and the way we structure our school systems has caused us to reward people for quickly having the answer to a question rather than saying, wait, is that the right question? Or rather than saying, is there more than one answer?

We’ve got behaviors like that, that have been entrenched since middle school? Before?

You’re not going to change those overnight, right? This is the whole thing I have about problem framing and solving, right? And, Roger sort of describes it as scientific method, analytical, but we’ve trained a generation of people and then we rewarded them for those behaviors.

It’s not going to change if we’re just talking at this big level about everything. It’s going to change because we model behavior changes ourselves. So when I teach not just product managers, but executives, this is what I talk about. Say, you can’t ask the people in your organization to do ABC if you’re not acting that way yourself. But they don’t even know that they’re acting that way. So they need to be, in effect, called out on it in a nice way, right?

So I would literally, at HP, my boss would do stuff in a meeting and I would walk up to him afterwards and I’d say, don’t ever say those words again, because here’s how they are heard. Don’t ask people to do X. Suggest that they do Y. Or, right, like I was trying to do literal behavior change work with him in order to get some of these changes in behavior, so it’s, you know, it takes systems thinking. It’s a big systems problem.

This is not going to get undone overnight. I’m sure you’ve done this in your coaching. How many times do you have somebody, you know, you say, oh, well, you should be customer centric about that. And three seconds later, they’re telling you what feature they want to put in the product,

Jesse: Right.

Sara: And you go, what problem is that solving? And they’re like, Oh, it’s hard. It’s hard.

Jesse: Yeah, so it’s interesting because it suggests there are several layers of the kind of culture change that’s really needed in organizations in order for design to genuinely deliver on its value proposition, because you’ve got to evolve the way that the culture engages with decision making. But you also have to evolve the way that the culture engages with problem framing and problem solving…

On Being Customer-Centric

Sara: And measuring itself. And, right?

The reward systems. It all has to align around achieving… That’s why the customer centricity thing, there’s actually not as much academic literature on that as you might think. It’s kind of surprising. If you say, Oh, are you a customer centric CEO? Well, of course I am. I talk to customers all the time. Like about what? Oh, well, that do they like the curvature of this or did they like this feature?

No, I don’t want you to talk to them about that. I want you to talk to them about their lives, about the problems they have about, right? We don’t have those kinds of conversations, but we think that because we shared a roadmap with the customer that we’re customer centric, right? It’s not. It’s not.

But if you don’t get down to that level, and they’re just going to go, well, we are customer centric. And I work with some large companies where they’ll, “Oh, we’re customer centric.” I go, okay, write an interview guide to go learn something about your customer. And we’ll get specific about X kind of problem. And what do they write? “What’s the last product you bought from us? Did you like that product?” “What do you…” right?

Like they’re not able to get into a mindset of learning, not selling. Of being present with a problem as opposed to with the solution, so it’s incredibly hard for them, but they think that because they’re talking, I mean, particularly in a complex B2B large technology kind of company, they are out talking to customers all the time, but they’re mostly talking about the features they’re developing. Not about the problems those features are aiming to solve.

Peter: More proxies.

Sara: Right. you know, to Jesse’s point, why is it so important that this change now? And you probably read Experience Economy…

Jesse: mm-hmm.

Sara: Pine and Gilmore stuff.

Jesse: Mm-hmm.

Sara: I really like the trajectory it lays out, which is to say, Commodities, the only differentiation is price, right? So coffee beans the value is can I get them at the lowest cost?

Then I make goods, grind up the coffee beans, put them in a can. I’ve added some value to the end customer. Services, I’ll make the coffee for you and serve it to you at Denny’s. Now I’ve added more value to the customer. Staging customer experiences. Now I’m, you know, at the Starbucks level of coffee where it’s the third place, there’s more going on than the coffee.

They’re now writing another book on guiding transformations. It used to be that most of the companies I taught were somewhere between goods and services. Now, most of them are trying to migrate to designing experiences. And very recently, people are starting to talk about guiding transformations. I cannot do experiences and transformations well, unless I truly understand what I’m trying to help my customers accomplish.

Jesse: Mm.

Sara: So, the trajectory we’re on, that trajectory is true. That increases differentiation and personalization, therefore increasing my opportunity to capture value, right?

So there’s an economic return to increasing the value I provide to the customer. Then we have no choice but to go through the cultural shift that you’re suggesting, Jesse.

And that’s kind of the broader argument that I’m trying to get people into, like, where do you want to be on this spectrum? You know, companies also talk about, well, hmm, sometimes we went to goods and then they got commoditized. So we went backwards. Oh, huh. I don’t know that I want to do that very often. Sometimes I might have to. But how do I always have my eye on the possibility that I might add additional value and differentiate in some way.

Embracing AI

Sara: Now, why is that being driven so fast? Because AI creates all this possibility to create, to stage experiences, to guide transformations.

But if I’m looking at it totally internally, I’m just going, how do I get AI into my product? So what’s everybody flocking to these days? Courses on how to put AI into their product. Rather than asking the question, how do I augment my ability to stage customer experiences or guide transformations?

AI will spit out a customer journey map for you, which, this is another place we talked earlier about, how do I get people to do this? I don’t have to spend time on this. It used to be I had to go interview a bunch of people. By the way, I’m not saying not to spend time with real customers. That’s a whole other conversation. But, but it used to be that I had to spend months to get an as is customer journey map, and now I can get one in less than a minute.

Are they always right? Are they always complete? No, they’re not. But in my experience in the last year or two, they’re pretty good, and that means I’ve had a lot of people try out prompts to get a customer journey map, and they can get a pretty good starting point.

Ah, if I can start there, now I’m playing a whole different game, because I don’t have to say, I don’t have time to blah, blah, blah, blah, right?

I can start somewhere. I can get a list of jobs to be done. What’s the customer support engineer trying to get done when resolving a customer issue? Here’s all the personal jobs to be done. Here’s all the professional jobs to be done. Do I still have to go play with that and say which of those jobs to be done would I like to support? What does my product support now? What are my competitive products support? Yes. But now I can do it with, more real detail.

Now, none of that is quantitative, right? You’re talking about a qualitative list of jobs to be done. I might try to do a quant evaluation, like how do I match up to my competitors on the ability to do those jobs for customers. That’s why I want a partnership with data science people.

Now I’ve got to get creative. And how do I take that existing experience and make it a whole lot better? So you’ve also got a whole bunch of questions about what are the roles of designers in a world when some of the artifacts that they’ve created historically, I can create pretty quickly now a first draft of. But now, now you’re in a cycle. Now I can go in with a first draft, I can say, you don’t have to pay me for six months of work…

Jesse: Well, there’s a catch in there though, right? Because the AI can make the journey map for you, but it can’t sell it to the executives for you. You’ve got to be the one to make the case and to frame the deliverable in some larger meaning, which comes back to this quant versus qual thing that we’ve been talking about.

But it also comes back to the notion of culture change, and the notion of organizational evolution that is a part of this. It is very common for design leaders to see themselves as organizational changemakers, as champions of a different way of thinking about decisions, a different way of thinking about prioritization, a different way of thinking about how you go to market as a business.

And you know, what I’m hearing reinforced in what you’re saying is that, like, you’ve got to really not just bring the data, but you’ve also got to bring the persuasion on a human level, to bring people around to this mindset as you describe it. The challenge that I hear from my clients is they feel like as design leaders, they are not sufficiently empowered to actually drive the mindset that is required to create design success. And again, coming back to the fact that you are not a design professor. You are teaching product managers. I find myself wondering about those design leaders who are asking, Am I even in the right place to create what I see myself creating in this organization as a design leader, or do I need to be somewhere else in order to actually accomplish these outcomes?

Create Your Own Power

Sara: Let me first say it is not just designers who say that. Product managers say that, you know, lots of people, so that’s not an uncommon concern. Part of the challenge we have is that the world is moving way faster, particularly since COVID, and we hit the knee of the technology change curve. The world has been upended. The people who are running the companies today have no experience in this new world. I would guess if you talk to some number of them, they’re terrified because…

Jesse: hmm.

Sara: …they don’t have the tools, the sense of what can happen. And so we’ve got real conflicts between the top of the organization and the people doing the work.

And the people who are doing the work often know more, but are feeling disempowered and I think part of it is this challenge of things are moving so fast, you can kind of get it. Even at HP when things weren’t moving that fast, you know, those of us more junior grew up with more technology then the people running the company, and so for them to be making judgments, they actually didn’t have that empathy, if you will, for what was happening out in their customer’s world.

So, so this kind of goes back to, create the power for yourself. I mean, no-, nobody’s going to come to you and say, I empower you to change how I think, right? It just isn’t that way.

Now, you could be in an organization that you should leave. And I assume that you coaches help people figure that out. Like if you’re pushing rope uphill, you know, there’s a point at which, don’t do that anymore.

But I ran the change management team at Hewlett Packard, right? Nobody said you’re empowered to help the company improve its teaming, change its product development process, improve supply chain management. I figured out that those were things that needed to happen, and went around and found places that would experiment with me.

I don’t have to change the whole company at once. I just have to redesign one product, and show that redesigning one product increased acceptance, adoption, whatever it is, right. So change doesn’t have to be, oh, they don’t understand me. No, they’re probably not going to, until you find that– I’m gonna go back to my thing.

I didn’t get that they cared a lot about consolidating engineers, not about consolidating production, right? Boom, I was telling the wrong story because I wasn’t listening to my senior management team.

That’s the same thing at the Adaptive Path conference where, you know, we were having somebody pretend to be the CMO. The designers who were sitting in my group weren’t listening to what the CMO cared about and creating a connection, right? it also goes to the whole storytelling thing, right? We teach storytelling a lot in product management. Why? Because, as our storytelling faculty member says, when you open with data, people receive it with their dukes up. Where’d you get the data? How’d you analyze it? Why are you sure that’s the average?

Start with the Story, Then the Data

Sara: If you open with a story and then provide the data, it’s heard in a different way. Even in companies that have very data driven focus, and I think, increasingly, in the product management space, I was just with a bunch of them this morning on storytelling, and one of them said, when we have a product that’s languishing, I use storytelling to figure out whether I should just kill it or what I need to do to fix it.

Because you can’t tell a good story if you don’t have a good story to tell.

Peter: Hm.

Sara: So it’s in the structuring of the story that I’m testing… “Once upon a time, there was a customer who really wanted to and could not do so because…” If I can’t tell a compelling, right, problem, then I don’t have something. So I would link these things together, right? Like, how do I bring the qualitative side? And I’ll come with data. “Once upon a time, there was a customer who really wanted to do this, couldn’t do so because, and by the way, there’s 400, 000 of those…

Peter: right.

Sara: in the United States alone.” Okay, now I’ve got my data. But you can’t expect it to happen overnight. I mean, that’s thing that I think is so… it just doesn’t, it just doesn’t.

Peter: Right.

Sara: So I’m a big “guide on the side” rather than “sage on the stage” as an educational concept.

And so to do guide on the side work, you’re gonna have a classroom that has tables and chairs, not the amphitheater style, right?

And so I used to go to these meetings about our new buildings and I’d be like, no, we’ve got to have flat classrooms. And it got to the point where I was in some meeting and it’s like 10 minutes into the meeting. So he goes, “Sara, you haven’t brought up the space problem yet.” So that’s what happens, right? Mean, to me, that’s what you have to sort of, with humor, be a bit of a pain in the neck, you know? Oh yeah, what problem is that solving? Oh yeah, what problem?

Pretty soon you start to hear it back again and they’re not even attributing it to you, which is the other part. You can’t take credit for it, right? Because, you’re just trying to get them to understand something and eventually it’ll click and then they’ll say it in their own terms and you’ll hear it come back. You go, well, I’ve said that. That’s right.

Peter: You’ve mentioned your work with product managers. Sounds like you’re teaching them working with maybe dozens, if not hundreds of them over the course of year. And I want to take advantage of your distinct kind of opportunity and perspective that that gives you, and I have a question which is, What are you hearing from them? What are the patterns you’re seeing in product management and with product managers? What are the things that are keeping them up at night or what stage are we at with product management?

I’m, just kind of curious as you look at the world of product management from your point of view across dozens if not hundreds of people from a lot of different enterprises. What have you witnessed? What insights do you have that could be of interest to people who are working with product managers and who don’t necessarily understand the challenges that they’re facing?

Sara: By the way, we have some number of designers who show up for the product management program. So it isn’t just product managers, we have fair number of technical or, engineering managers who show up as well. And I think it’s in part because they would like to understand a little bit better how business cases are being made for this.

So, first, let me say, because I’ve always been in this boat as long as you’ve known me, or maybe even before, I’ve always taught human centered or customer centered development, right? That’s what we taught at Berkeley in the MBA programs.

And that’s really how we anchor the product management program. So, yes, we teach segmentation, targeting, positioning, pricing strategy. But the Trojan horse, if you will, is it’s all customer centric.

So the first thing I would say is that companies used to push back heavily on that, let’s say, 15, 20 years ago. A lot of the stuff you’re saying, Oh, no, we can’t do that. My boss would never let me do that, etc. There’s a lot less of that now. In fact, it’s pretty rare. I can credit design thinking with that, if you like, that there’s been enough conversation and agenda around some of these first principles of customer centric work, and adoption of things like rapid prototyping, although aside, we teach risky assumption testing rather than minimum viable product, because there are often assumptions you can test without a minimum viable product.

So rapid prototyping has become minimum viable product instead of risky assumption testing.

Jesse: Right.

Sara: So being clear about what are the risks associated with the idea that I have for creating a new customer experience, and how might I test them, could lead you to, I’ll test them through a minimum viable product, but it doesn’t have to lead you there. But most companies don’t get that until you really push them.

This is another one of those things where you can push them at a micro level. It’s like, what do you want to learn from making that minimum viable product? And often they can’t even tell you, because it’s just built into their agile process that they all spit out minimum viable products.

These are all the little things that, if you could get that mindset to shift, then you end up in a different place. So that has definitely changed. And there’s a lot more acceptance of customer centric thinking on the part of product managers.

The second thing that’s really big is that, of course, the field of product management, as you probably well known, has grown significantly. So there are a lot of product managers now. I don’t remember the numbers on LinkedIn, but it’s grown significantly. That includes not only the classic externally facing product manager, but it also includes a lot of internally facing product managers. So they might manage a platform on which products are built for that external customers or users.

So, product management is sort of propagated in a lot of ways, which is probably why you see it encroaching more and more on what designers do, because, theoretically anyway, the product manager’s job is to bring together the customer with the engineering community, to make stuff happen, right? It’s more complex than that, but they’re supposed to be a representation of the customer.

I would say in big companies, most of them have CX/UX design group, but it’s not always accessible. So you can’t get your project in there unless it’s prioritized in some way. So not everybody in the organization feels like they can get to those capabilities.

B, very often what is produced is a bullet point list of things to build. So it’s not bringing alive the existing customer experience or the experience they want to build. Dorothy Leonard at Harvard used to call this empathic design. That I have in the guts of the engineers empathy for the customer.

Well, if the CX design group is doing all that, but it’s not presented to the product manager in a form that allows them to use it to create that empathy, right, then we’re missing a connection.

So the product managers are not really doing that kind of design work. Like many of them don’t interview customers. Somebody else does that. Which is too bad because they should be out in front of customers themselves. So this becomes the place where your question gets messy in the sense of, first of all, who does what, but then, where’s the connection between the two?

I had a PhD student write an entire thesis about this question of how I present the results of design research to the people that will execute the product and not lose something in the process. So this could be another reason why designers feel left out, because maybe the bridge, there’s a gap, like something’s jumping across from the customer experience group into the PMs, but it’s not really a connected thing, like, could they bring the PMs on interviews with them? They probably should once in a while, right? So the PMs can actually see the customers. Do they help digest the data with the PMs? Probably not, right?

What’s Next for Design

Jesse: So, I wonder what all of this suggests for the future of design. And I’m curious what you think and what you see from your vantage point, in terms of what’s next for where design is going in organizations.

Sara: Yeah, designers have very particular skill sets that are not generally present in the rest of the organization. For example, the ability to conduct a good ethnographic interview with a customer or user. The ability to create representations of an offering, whether it’s an experience, a service, whatever. There are capabilities that don’t exist elsewhere in the organization.

Jesse: So what you’re saying is they have a very particular set of skills.

Peter: Ha!

Sara: That are important, and the question is, how do they get embedded in the organization, right? How and where? So, do product managers really understand… And this goes back to, in 1993, a woman who was a fashion designer in New York came to get her MBA, and after she took my operations class, she came to me, and she said, shouldn’t business people know more about design? And I said, sure, start a class.

So she started a full on three unit class called Design as a Strategic Business Issue. And, Davis Masten, Jerry Hirshberg, Sara Little Turnbull, Bill Moggridge, right? Like, name people in that world. They came with their 35mm slide trays, and they showed what designers do. And it was eye opening and mind blowing for the MBA students.

That class isn’t taught anymore. Partly because Design thinking came along, and so we started teaching design thinking, and now we’re not teaching what those critical skills that designers bring are and what they produce, right? I mean, there’s all kinds of stuff that we talk, oh, in design thinking, we do visualization because we put post its on the wall. Well, that’s not the same as, you know, my friend Michael Berry sketching what a new drive-through would be for a fast food company while everybody’s talking about it, right?

And so part of the challenge then becomes, how do I make clear what those unique skills of designers bring to an organization. Then I would also ask the question, if I centralize something, then it’s not part of the fabric of the organization. So do I need a UX designer on every single team? Like does every product manager need one? I don’t know. Do they need at least consulting with one once a week? Maybe? I’m throwing this out there because I don’t know if anybody’s even asking those questions, probably the two of you.

But, how do I bring that skill set to bear when it’s most valuable and needed? Right now, my sense is that it’s separate out there for a lot of big companies. It’s somewhere up here and maybe I’ll get access to it, right.

And by the way, rapid testing of things is primarily A/B testing that’s owned by engineers, or MVPs that are owned by engineers. So this capability to do early testing is not embedded in the organization.

But the agile feature mill is a fast cycle, but I lose the front end then. Like, product managers works gotten sucked into the agile feature mill. Well, ditto with any front end design work, right? So the question is, where do you want to embed those capabilities, as opposed to thinking of them as people?

I know they’re people and they’re wonderful people, but really thinking about what is the capability set? What are the jobs they can help with? And where do I need to have those jobs done in an organization? And then how do I make that happen on the front lines, as opposed to in a big, you know, hand wavy way, which we kind of get stuck in, don’t we, when we talk about design?

Jesse: Dr. Sara Beckman. Thank you so much for being with us.

Sara: It has been my pleasure.

Sara where can people find you?

I’m reachable on LinkedIn. I just would ask if somebody reaches out on LinkedIn that they tell me why they know me.

Peter: Not just a message that says hello.

Jesse: Mention the podcast. Mention the podcast.

Sara: Mention the podcast. There you go. and I’m also available by email, which is just Beckman at Berkeley dot edu.

Peter: Awesome. Thank you so much.

For those who have stuck around, we have a special treat for you. We have revamped our website at Finding Our Way dot Design, and we have also launched a new online community through Discord called The Way Station. So go to Finding Our Way dot Design. Click on the link that says The Way Station, and you’ll see a link to our new Discord server.

Hope to see you there.

Jesse: For more Finding Our Way, visit findingourway. design for past episodes and transcripts. You can now follow Finding Our Way on LinkedIn as well. For more about your hosts, visit our websites, petermerholtz. com and jessejamesgarrett. com. Peter recently launched The Merholz Agenda, his semi weekly newsletter. Find it at buttondown.com slash petermerholz. And if you’re curious about working with me as your coach, book your free introductory session at JesseJamesGarrett. com slash free coaching. If you’ve found value in something you’ve heard here today, we hope you’ll pass this episode along to someone Else who can use it. Thanks for everything you do for others, and thanks so much for listening.