🚀 Faster, Please! — The Podcast #22

"You can see the computer age everywhere but in the productivity statistics," said Nobel laureate economic Robert Solow in 1987. A decade later, the '90s productivity boom was in full swing. Likewise, it took decades for electrification to have an impact on productivity growth in the early 20th century. Today, artificial intelligence can write a coherent paragraph or generate an image from a simple prompt. But when will AI show up in the statistics, boosting productivity and then economic growth? Avi Goldfarb joins Faster, Please! — The Podcast to discuss that question and more.

Avi holds the Rotman Chair in Artificial Intelligence and Healthcare at the University of Toronto's Rotman School Of Management. He's also co-author, along with Ajay Agrawal and Joshua Gans, of 2022's Power and Prediction: The Disruptive Economics of Artificial Intelligence.

In This Episode

* Prediction at scale (1:34)

* How AI has transformed ride hailing and marketing (5:37)

* The potential for “system-level” changes (11:26)

* When will AI show up in the statistics? (16:12)

* The impact of ChatGPT and DALL-E (19:46)

Below is an edited transcript of our conversation.

Prediction at scale

James Pethokoukis: What this book is about—and then you can tell me if I've gotten it horribly wrong—this is a book about machines making predictions using advanced statistical techniques. 1) Is that more or less right? And 2) why is that an important capability?

Avi Goldfarb: That's more or less right. The only place where I [would offer] a little correction there is, the reason we're talking about artificial intelligence today is almost entirely due to advances in computational statistics. Yes, it is just stats and that sounds kind of unexciting. But once we have prediction at scale, it can be really transformative to all aspects of business in the economy. There's a reason why we're calling computational stats “artificial intelligence” and we didn't use to.

Prediction at scale. That's a great three-word description. Probably why you used it. To what extent is that now happening? The name of the book is Power and Prediction: The Disruptive Economics of Artificial Intelligence. Is this prediction at scale already disruptive to some degree or is it, will be disruptive?

The technology, for the most part, is pretty close to there, in the sense that we can do prediction at scale because we have the data and we have computational power to do all sorts of amazing things. For the most part, it hasn't been disruptive yet. And it hasn't been disruptive yet, just because we have the technology doesn't mean we know how to use it well and we know how to use it productively in our processes and systems in order to get the most out of it.

Are there sectors currently doing this, but they're not doing it well yet? It’s in a variety of sectors, but not enough companies doing it?

Lots of companies are already using these machine learning tools, but they tend to be using them for things they were already doing before. If you had some prediction process to predict, if you're a bank, whether somebody's going to pay back a loan. In the very old days you'd have some human, the loan officer, look the customer up and down and go with their gut. And then, starting in the 1960s and especially in the ‘90s and beyond, we started to use scoring rules, partly your credit score and partly other things, to get a sense of whether people are going to pay them back. And so we were already doing a prediction task done by a machine. And now increasingly we're using these machine learning tools. We're using what we're calling AI, over the past five to 10 years, to predict whether people are going to pay back a loan. We're seeing those kinds of things all over the place, which is: You had some prediction, maybe you’ve used even a machine prediction before, and now we're using machine learning. We're using AI to make those predictions a little bit better. Lots of companies are using that.

That sounds incremental. That sounds like an incremental advance.

It's absolutely an incremental advance. We call these point solutions, which is, you look at your workflow, you identify something that a human is doing. You take out that human; you drop in a machine. You don't mess with a workflow because it's always easier to do things when you don't mess with a workflow. The problem is, when you don't mess with a workflow, there's only so much gain you can get. We've seen AI-based point solutions, prediction point solutions, all over the place. We haven't seen real transformation in very many industries. We've seen it in a couple. We haven't seen it in very many industries because real transformation requires doing things differently.

How AI has transformed ride hailing and marketing

Do you think that it has happened in one or two industries that you think would actually meet that bar of transformational? Can you give me an example?

Absolutely. If you wanted to be a cab driver in the city of London 20 years ago, or even today, it takes three years of schooling. Learning to navigate those streets is really, really hard. And especially learning to navigate and predict where the traffic is going to be is really, really hard. And so there is a really rigorous process to screen people to be taxi drivers. In the US 30 years ago, there was something like 200 or 300 taxi drivers in the whole country. About 15 years ago, two technologies came about. The first one being digital dispatch, which is essentially tools for drivers to find riders, sometimes through prediction and sometimes through other tools. And then the second part was what's been disruptive with respect to that three years of schooling in the city of London, which is prediction tools for navigating a city. This is your GPS system.

In the early days, many people selling digital dispatch and navigational predictions were selling them into professional driving companies, into taxi companies. “Hey, your taxi drivers can be 15 percent more efficient if they know the best route at this time.” That's what we call a point solution. You’re already doing this, you take out some part of the human process, you drop in a machine, and you do it a little bit better. A couple of companies realized that digital dispatch combined with navigational prediction could create an entirely new type of industry. And this is the ride-hailing industry led by Uber and Lyft and others. That's a totally new kind of way to do personal transportation that made millions of amateur drivers as good as professional because they could navigate the city and find riders.

Example number one is the taxi industry. Personal ride-hailing, for lack of a better word, has been transformed partly through digital and really those maps are important—and a big part of those maps is machine learning tools and figuring out where the traffic is, etc. So industry number one.

Industry number two is advertising. I don't know if you've seen the TV show Mad Men. That was really how the advertising industry operated well into the ‘90s. Maybe not the soap opera aspect of it. Maybe, maybe not. I don't know. But the idea that there's a lot of wining and dining and charming people to convince them to spend millions of dollars on an ad campaign. And whether a campaign worked or not was largely based on gut feel. And which kinds of customers you targeted and which TV show and which magazine, all of that was priced based on intuition and not much else.

Digital advertising came along in the late 1990s, and the first ways we thought about digital advertising was that it was like the magazine industry. So instead of advertising in People magazine, you're going to advertise on Yahoo using the exact same processes you did in People magazine. There was a rate card and it was going to be so many dollars per thousand users. And if you were doing general search, it might be $10, and if you're looking for real estate, it might be $50. And that's exactly how the magazine industry was priced. Some magazines were more than others based on readership and topic. And it was all based on personal selling, intuition, deals, etc.

Then people realize that digital advertising created an opportunity to predict who the user was, who might see your ad. A user arrives at a publisher and an ad needs to be served, and you can predict who that user is and what they might want and when they might want it. Based on those predictions, rather than just do the magazine industry old way of doing things, you can now serve the right ad to the right person at the right time. Starting around 2000, there were all these innovations in online advertising that led to an industry that today looks almost nothing like the industry that you saw in Mad Men. Every time a user goes to a website, there is a real-time auction, in fractions of a second, between, in effect, thousands of advertisers for that user's attention. And there are all these intermediary steps, lots and lots of intermediaries—largely led by Google, but some other players that complement Google in that process—to create an entirely new kind of ad industry. The ad industry has had a system-level change because we can now predict, for a given impression or given user who's looking at a page, what they might want and when they might want it. Predictions changed the industry.

The potential for “system-level” changes

How confident are you that this technology is powerful enough that we'll see system-level changes across the economy? That this is a general-purpose technology that will be significant? And do we have any idea what those changes will be, or is it, “They'll be big, but we don't know exactly what they are.”

The technology itself is pretty extraordinary. And so in lots and lots of contexts, I'm pretty confident the technology's going to get there. There are some constraints on it, which is that you need data on the thing you're trying to predict in order for the predictions to work. But there are lots and lots of industries where we have great data. The technology barriers, I think, are being overcome. In some industries faster than others, but they're being overcome in lots and lots of places.

That's not the only barrier. The technology is barrier number one. Think of an industry that I'm particularly excited about the potential of the technology, which is healthcare. Why is it so exciting for healthcare? Because diagnosis is at the center of how healthcare operates. If you know what's wrong with somebody, it's much easier to treat them, it's much less costly to treat them, and you can deliver the right treatment to the right person at the right time. Diagnosis, by the way, is prediction. It wasn't obvious, the way we thought about that in the past. But really, what it is, it can be solved [with] statistical prediction by using the information you have, the data on your symptoms, to fill in the information you don't have, which is what's actually causing your symptoms. If you do a Google Scholar search for something like “artificial intelligence healthcare,” you'll get a few million hits. There are lots of people who've done research producing AI for diagnosis. The technology, in many cases, is there. And in lots of other cases, it's pretty close.

That doesn't mean it's going to transform healthcare. Why not? What's an AI doing diagnosis? They're doing a thing that makes doctors special. Yes, a good doctor in their workflow does all sorts of other things — they help patients navigate the stress of the healthcare system, they provide some treatments, etc. — but the thing that they went to school for all those years for, and for many of them the thing that they have that nurses and pharmacists and other medical professionals don't, is the ability to diagnose. When you bring in machine diagnosis into the healthcare system, that's going to be very disruptive to doctors. There are lots of reasons why, then, doctors might resist. First, they might be worried about their own jobs. Second, they might just not trust the machines and believe they're good enough. Because [in] the medical system doctors are a core source of power—they help determine how things work—they're going to resist many of the biggest system-level changes from AI-based diagnosis.

And so you may have regulatory barriers, you may have organizational incentive barriers, and you may have barriers from the individual people on the ground who sabotage the machines that are trying to replace them. All of these are reasons — even if the technology is good enough — that AI in healthcare may be a long way away, even though we can see what that vision looks like. In other industries, it might be closer. In lots of retail contexts, you’re trying to figure out who wants what and when — Amazon's pretty good at that in lots of ways — and in-store retailers can do that too. And so there are reasons to think that disruption in many retail industries will come faster.

I just want to be a little careful here. I see the technology is there. There are some barriers on the technology side. If the payoff is big enough, I think most of the technology-related barriers can be overcome. To give you a sense of this: We hear a lot something like, “We don't want to do AI in our company because it's just so difficult to get the data organized and get the right data to build those predictions.” Well, yeah, it's difficult. But if the payoff is going to be transformative to the company and make the company millions or billions of dollars, then they'll spend thousands or millions in order to make it happen. And so a lot of the challenges aren't tech specific. They're incentives and organization based.

When will AI show up in the statistics?

I think of the classic Paul David paper about the dynamo. It took a while before factories used electricity, and they actually had to redo how the factory was designed to get full productivity value. And you say that we are sort of in the “between times.” And that makes me think of a classic Solow paradox: We see computers everywhere but in the statistics. He said that in ’87. Are we, like, in the 1987 period with this technology? Or are we now in the late ‘90s where it's starting to happen and the boom is about to begin?

I think we're in the early ‘80s.

Not even the late ‘80s?

He said that in 1987. By 1990 it was showing up in the data. So he just missed it.

[We’re in the early 1980s] in the sense that we don't quite know what the organization of the future looks like. There are reasons to think for many industries it might take a long time, like many years or decades, for it to show up in the productivity stats. While I do say we're in the early ‘80s because we haven't figured it out yet, I'm a little more optimistic that maybe it won't be 30 years to really have the impact. Mostly because we just have the lessons of history. We know from past technologies, and business leaders know from past technologies, electricity and the internet and the steam engine and others, that it requires some system-level change. And we now have the toolkits to think through, how do you build system-level change without destroying your company?

When electricity was diffusing in the 1890s, there wasn't really any idea that this might take 40 years to figure out what the factory of the future looks like. It just wasn't on anybody's mind. The management challenges of redesign were unstudied, and there was no easily accessible knowledge to figure that out. Jump forward to the ‘80s and computing: Again, we hadn't even learned the lessons of electricity back then. Paul David's paper came out in 1990. It was a solution to the Solow paradox.

But since then, we have a much better understanding of what's required for technological change. There has been decades of economics literature Erik Brynjolfsson, Tim Bresnahan, Paul David, and others. And there's been decades of management literature taking a lot of those ideas from econ and trying to communicate them to a broader audience to say, “Yes, it's hard. But doing nothing can also be a disaster. So being proactive is useful.” Then there's another piece about optimism here, which is that the entrepreneurial ecosystem is different than it used to be. And we have lots and lots of very smart people building tech companies, trying to make the system-level change happen. And that gives us more effectively more kicks at the can to actually figure out what the right system looks like.

The impact of ChatGPT and DALL-E

ChatGPT and these text-to-image generators like DALL-E, are these significant innovations that can cause system change? Or are they toys that can't figure out how many arms people have and are able to produce B-level middle school essays?

They're both. What do I mean by that? The technology is incredible. What ChatGPT can do and DALL-E can do is really, at least to me, it's amazing. Especially what ChatGPT can do. It's much better than I… That came much faster than at least I thought it was going to come. When I first saw it, I was blown away. So far it's a toy. So far, most applications have been “Hey, isn't this cool? I can do this kind of thing.” In a handful of places, it’s moved beyond a toy to a point solution. Joshua [Gans], Ajay [Agrawal], and I wrote a piece in HBR. We drafted it out, and rather than reread it and edit it 60 times like we normally do, we sent it into ChatGPT and said, “Write this in a way that's easy to read.” And it did. We had to do some final edits afterwards. But like, we are already doing the same thing. It made a piece of our workflow a little bit more efficient. Point solution.

A lot of the talk here in universities, “Uh-oh, we have to change the way we do final exams because ChatGPT can write those exams for our students.” Sure. But that's really not thinking through the potential of what the technology can do. What we've seen so far are toys and point solutions, but I do see extraordinary potential for system solutions in both. Both DALL-E and ChatGPT, and all these generative models. ChatGPT, if you think about it, what does it do? One thing it does is it allows anybody to write well. Like I told my students, you no longer have an excuse to write a bad essay with terrible grammar and punctuation that's not structured like a five-paragraph essay. No excuse anymore. It used to be, okay, maybe there's an excuse because there was some time crunch and you had other things due. Or your language skills — you're a math person, not an English person. No excuse anymore. ChatGPT upskills all those people who are good at other things but whose opportunities were constrained by their ability to write. So what's that new system? I don't know. But there are a lot more people around the world who are bad at writing English than are good at writing English. And if now everybody is a B high school-level student, able to write an essay or able to write well in English, an email or whatever it might be, that's going to be amazing. We just have to figure out how to harness that. We haven't yet.

You’ve sort of given us a potential timeframe, broadly, for when we might see this in the data. When we see it in the data, how significant do you think this technology can be? What is, do you think, the potential impact once you can find it in the data, the productivity growth, which is kind of the end goal is here?

That’s a great question. Let me reframe it and say, the thing I'm worried about is that it won't reach its potential. A lot of people are worried about the impact of AI on jobs and what are people going to do if machines are intelligent? Jason Furman attended our first Economics of AI conference. This was in 2017. He was formerly chair of Obama's Council of Economic Advisors. And the thing I'm worried about is that there's not going to be enough AI. The productivity booms that we've had in history from way back to the steam engine and then electricity and then the computer age and the internet have been driven by system-level change, where we've figured out how to reinvent the economy. And that's led to sustained productivity growth: first the steam engine at 0.5 percent and then maybe 1 percent with electricity and then 2 percent after the war or more. I don’t know what the number is going to be. I know you wanted me to give you a number. I don’t know what the number's going to be. But this technology has potential to be like all those others, assuming we figure out what that system-level change looks like and we overcome the various sources of resistance.

To sum it up, your concern is less about, can we solve the technical problems, versus, will society accept the results?

Exactly.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit fasterplease.substack.com/subscribe