How incumbents get dislodged by tech shifts

My fellow pro-growth/progress/abundance Up Wingers,

For most of history, stagnation — not growth — was the rule. To explain why prosperity so often stalls, economist Carl Benedikt Frey offers a sweeping tour through a millennium of innovation and upheaval, showing how societies either harness — or are undone by — waves of technological change. His message is sobering: an AI revolution is no guarantee of a new age of progress.

Today on Faster, Please! — The Podcast, I talk with Frey about why societies midjudge their trajectory and what it takes to reignite lasting growth.

Frey is a professor of AI and Work at the Oxford Internet Institute and a fellow of Mansfield College, University of Oxford. He is the director of the Future of Work Programme and Oxford Martin Citi Fellow at the Oxford Martin School.

He is the author of several books, including the brand new one, How Progress Ends: Technology, Innovation, and the Fate of Nations.

In This Episode

* The end of progress? (1:28)

* A history of Chinese innovation (8:26)

* Global competitive intensity (11:41)

* Competitive problems in the US (15:50)

* Lagging European progress (22:19)

* AI & labor (25:46)

Below is a lightly edited transcript of our conversation.

The end of progress? (1:28)

. . . once you exploit a technology, the processes that aid that run into diminishing returns, you have a lot of incumbents, you have some vested interests around established technologies, and you need something new to revive growth.

Pethokoukis: Since 2020, we’ve seen the emergence of generative AI, mRNA vaccines, reusable rockets that have returned America to space, we’re seeing this ongoing nuclear renaissance including advanced technologies, maybe even fusion, geothermal, the expansion of solar — there seems to be a lot cooking. Is worrying about the end of progress a bit too preemptive?

Frey: Well in a way, it’s always a bit too preemptive to worry about the future: You don’t know what’s going to come. But let me put it this way: If you had told me back in 1995 — and if I was a little bit older then — that computers and the internet would lead to a decade streak of productivity growth and then peter out, I would probably have thought you nuts because it’s hard to think about anything that is more consequential. Computers have essentially given people the world’s store of knowledge basically in their pockets. The internet has enabled us to connect inventors and scientists around the world. There are few tools that aided the research process more. There should hardly be any technology that has done more to boost scientific discovery, and yet we don’t see it.

We don’t see it in the aggregate productivity statistics, so that petered out after a decade. Research productivity is in decline. Measures of breakthrough innovation is in decline. So it’s always good to be optimistic, I guess, and I agree with you that, when you say AI and when you read about many of the things that are happening now, it’s very, very exciting, but I remain somewhat skeptical that we are actually going to see that leading to a huge revival of economic growth.

I would just be surprised if we don’t see any upsurge at all, to be clear, but we do have global productivity stagnation right now. It’s not just Europe, it’s not just Britain. The US is not doing too well either over the past two decades or so. China’s productivity is probably in the negative territory or stagnant, by more optimistic measures, and so we’re having a growth problem.

If tech progress were inevitable, why have predictions from the ’90s, and certainly earlier decades like the ’50s and ’60s, about transformative breakthroughs and really fast economic growth by now, consistently failed to materialize? How does your thesis account for why those visions of rapid growth and progress have fallen short?

I’m not sure if my thesis explains why those expectations didn’t materialize, but I’m hopeful that I do provide some framework for thinking about why we’ve often seen historically rapid growth spurts followed by stagnation and even decline. The story I’m telling is not rocket science, exactly. It’s basically built on the simple intuitions that once you exploit a technology, the processes that aid that run into diminishing returns, you have a lot of incumbents, you have some vested interests around established technologies, and you need something new to revive growth.

So for example, the Soviet Union actually did reasonably well in terms of economic growth. A lot of it, or most of it, was centered on heavy industry, I should say. So people didn’t necessarily see the benefits in their pockets, but the economy grew rapidly for about four decades or so, then growth petered out, and eventually it collapsed. So for exploiting mass-production technologies, the Soviet system worked reasonably well. Soviet bureaucrats could hold factory managers accountable by benchmarking performance across factories.

But that became much harder when something new was needed because when something is new, what’s the benchmark? How do you benchmark against that? And more broadly, when something is new, you need to explore, and you need to explore often different technological trajectories. So in the Soviet system, if you were an aircraft engineer and you wanted to develop your prototype, you could go to the red arm and ask for funding. If they turned you down, you maybe had two or three other options. If they turned you down, your idea would die with you.

Conversely, in the US back in ’99, Bessemer Venture declined to invest in Google, which seemed like a bad idea with the benefit of hindsight, but it also illustrates that Google was no safe bet at the time. Yahoo and Alta Vista we’re dominating search. You need somebody to invest in order to know if something is going to catch on, and in a more decentralized system, you can have more people taking different bets and you can explore more technological trajectories. That is one of the reasons why the US ended up leading the computer revolutions to which Soviet contributions were basically none.

Going back to your question, why didn’t those dreams materialize? I think we’ve made it harder to explore. Part of the reason is protective regulation. Part of the reason is lobbying by incumbents. Part of the reason is, I think, a revolving door between institutions like the US patent office and incumbents where we see in the data that examiners tend to grant large firms some patents that are of low quality and then get lucrative jobs at those places. That’s creating barriers to entry. That’s not good for new startups and inventors entering the marketplace. I think that is one of the reasons that we haven’t seen some of those dreams materialize.

A history of Chinese innovation (8:26)

So while Chinese bureaucracy enabled scale, Chinese bureaucracy did not really permit much in terms of decentralized exploration, which European fragmentation aided . . .

I wonder if your analysis of pre-industrial China, if there’s any lessons you can draw about modern China as far as the way in which bad governance can undermine innovation and progress?

Pre-industrial China has a long history. China was the technology leader during the Song and Tang dynasties. It had a meritocratic civil service. It was building infrastructure on scales that were unimaginable in Europe at the time, and yet it didn’t have an industrial revolution. So while Chinese bureaucracy enabled scale, Chinese bureaucracy did not really permit much in terms of decentralized exploration, which European fragmentation aided, and because there was lots of social status attached to becoming a bureaucrat and passing the civil service examination, if Galileo was born in China, he would probably become a bureaucrat rather than a scientist, and I think that’s part of the reason too.

But China mostly did well when the state was strong rather than weak. A strong state was underpinned by intensive political competition, and once China had unified and there were fewer peer competitors, you see that the center begins to fade. They struggle to tax local elites in order to keep the peace. People begin to erect monopolies in their local markets and collide with guilds to protect production and their crafts from competition.

So during the Qing dynasty, China begins to decline, whereas we see the opposite happening in Europe. European fragmentation aids exploration and innovation, but it doesn’t necessarily aid scaling, and so that is something that Europe needs to come to terms with at a later stage when the industrial revolution starts to take off. And even before that, market integration played an important role in terms of undermining the guilds in Europe, and so part of the reason why the guilds persist longer in China is the distance is so much longer between cities and so the guilds are less exposed to competition. In the end, Europe ends up overtaking China, in large part because vested interests are undercut by governments, but also because of investments in things that spur market integration.

Global competitive intensity (11:41)

Back in the 2000s, people predicted that China would become more like the United States, now it looks like the United States is becoming more like China.

This is a great McKinsey kind of way of looking at the world: The notion that what drives innovation is sort of maximum competitive intensity. You were talking about the competitive intensity in both Europe and in China when it was not so centralized. You were talking about the competitive intensity of a fragmented Europe.

Do you think that the current level of competitive intensity between the United States and China —and I really wish I could add Europe in there. Plenty of white papers, I know, have been written about Europe’s competitive state and its in innovativeness, and I hope those white papers are helpful and someone reads them, but it seems to be that the real competition is between United States and China.

Do you not think that that competitive intensity will sort of keep those countries progressing despite any of the barriers that might pop up and that you’ve already mentioned a little bit? Isn’t that a more powerful tailwind than any of the headwinds that you’ve mentioned?

It could be, I think, if people learn the right lessons from history, at least that’s a key argument of the book. Right now, what I’m seeing is the United States moving more towards protectionist with protective tariffs. Right now, what I see is a move towards, we could even say crony capitalism with tariff exemptions that some larger firms that are better-connected to the president are able to navigate, but certainly not challengers. You’re seeing the United States embracing things like golden shares in Intel, and perhaps even extending that to a range of companies. Back in the 2000s, people predicted that China would become more like the United States, now it looks like the United States is becoming more like China.

And China today is having similar problems and on, I would argue, an even greater scale. Growth used to be the key objective in China, and so for local governments, provincial governments competing on such targets, it was fairly easy to benchmark and measure and hold provincial governors accountable, and they would be promoted inside the Communist Party based on meeting growth targets. Now, we have prioritized common prosperity, more national security-oriented concerns.

And so in China, most progress has been driven by private firms and foreign-invested firms. State-owned enterprise has generally been a drag on innovation and productivity. What you’re seeing, though, as China is shifting more towards political objectives, it’s harder to mobilize private enterprise, where the yard sticks are market share and profitability, for political goals. That means that China is increasingly relying more again on state-owned enterprises, which, again, have been a drag on innovation.

So, in principle, I agree with you that historically you did see Russian defeat to Napoleon leading to this Stein-Hardenberg Reforms, and the abolishment of Gilded restrictions, and a more competitive marketplace for both goods and ideas. You saw that Russian losses in the Crimean War led to the of abolition of serfdom, and so there are many times in history where defeat, in particular, led to striking reforms, but right now, the competition itself doesn’t seem to lead to the kinds of reforms I would’ve hoped to see in response.

Competitive problems in the US (15:50)

I think what antitrust does is, at the very least, it provides a tool that means that businesses are thinking twice before engaging in anti-competitive behavior.

I certainly wrote enough pieces and talked to enough people over the past decade who have been worried about competition in the United States, and the story went something like this: that you had these big tech companies — Google, and Meta, Facebook and Microsoft — that these were companies were what they would call “forever companies,” that they had such dominance in their core businesses, and they were throwing off so much cash that these were unbeatable companies, and this was going to be bad for America. People who made that argument just could not imagine how any other companies could threaten their dominance. And yet, at the time, I pointed out that it seemed to me that these companies were constantly in fear that they were one technological advance from being in trouble.

And then lo and behold, that’s exactly what happened. And while in AI, certainly, Google’s super important, and Meta Facebook are super important, so are OpenAI, and so is Anthropic, and there are other companies.

So the point here, after my little soliloquy, is can we overstate these problems, at least in the United States, when it seems like it is still possible to create a new technology that breaks the apparent stranglehold of these incumbents? Google search does not look quite as solid a business as it did in 2022.

Can we overstate the competitive problems of the United States, or is what you’re saying more forward-looking, that perhaps we overstated the competitive problems in the past, but now, due to these tariffs, and executives having to travel to the White House and give the president gifts, that that creates a stage for the kind of competitive problems that we should really worry about?

I’m very happy to support the notion that technological changes can lead to unpredictable outcomes that incumbents may struggle to predict and respond to. Even if they predict it, they struggle to act upon it because doing so often undermines the existing business model.

So if you take Google, where the transformer was actually conceived, the seven people behind it, I think, have since left the company. One of the reasons that they probably didn’t launch anything like ChatGPT was probably for the fear of cannibalizing search. So I think the most important mechanisms for dislodging incumbents are dramatic shifts in technology.

None of the legacy media companies ended up leading social media. None of the legacy retailers ended up leading e-commerce. None of the automobile leaders are leading in EVs. None of the bicycle companies, which all went into automobile, so many of them, ended up leading. So there is a pattern there.

At the same time, I think you do have to worry that there are anti-competitive practices going on that makes it harder, and that are costly. The revolving door between the USPTO and companies is one example of that. We also have a reasonable amount of evidence on killer acquisitions whereby firms buy up a competitor just to shut it down. Those things are happening. I think you need to have tools that allow you to combat that, and I think more broadly, the United States has a long history of fairly vigorous antitrust policy. I think it’d be a hard pressed to suggest that that has been a tremendous drag on American business or American dynamism. So if you don’t think, for example, that American antitrust policy has contributed to innovation and dynamism, at the very least, you can’t really say either that it’s been a huge drag on it.

In Japan, for example, in its postwar history, antitrust was extremely lax. In the United States, it was very vigorous, and it was very vigorous throughout the computer revolution as well, which it wasn’t at all in Japan. If you take the lawsuit against IBM, for example, you can debate this. To what extent did it force it to unbundle hardware and software, and would Microsoft been the company it is today without that? I think AT&T, it’s both the breakup and it’s deregulation, as well, but I think by basically all accounts, that was a good idea, particularly at the time when the National Science Foundation released ARPANET into the world.

I think what antitrust does is, at the very least, it provides a tool that means that businesses are thinking twice before engaging in anti-competitive behavior. There’s always a risk of antitrust being heavily politicized, and that’s always been a bad idea, but at the same time, I think having tools on the books that allows you to check monopolies and steer their investments more towards the innovation rather than anti-competitive practices, I think is, broadly speaking, a good thing. I think in the European Union, you often hear that competition policy is a drag on productivity. I think it’s the least of Europe’s problem.

Lagging European progress (22:19)

If you take the postwar period, at least Europe catches up in most key industries, and actually lead in some of them. . . but doesn’t do the same in digital. The question in my mind is: Why is that?

Let’s talk about Europe as we sort of finish up. We don’t have to write How Progress Ends, it seems like progress has ended, so maybe we want to think about how progress restarts, and is the problem in Europe, is it institutions or is it the revealed preference of Europeans, that they’re getting what they want? That they don’t value progress and dynamism, that it is a cultural preference that is manifested in institutions? And if that’s the case — you can tell me if that’s not the case, I kind of feel like it might be the case — how do you restart progress in Europe since it seems to have already ended?

The most puzzling thing to me is not that Europe is less dynamic than the United States — that’s not very puzzling at all — but that it hasn’t even managed to catch up in digital. If you take the postwar period, at least Europe catches up in most key industries, and actually lead in some of them. So in a way, take automobiles, electrical machinery, chemicals, pharmaceuticals, nobody would say that Europe is behind in those industries, or at least not for long. Europe has very robust catchup growth in the post-war period, but doesn’t do the same in digital. The question in my mind is: Why is that?

I think part of the reason is that the returns to innovation, the returns to scaling in Europe are relatively muted by a fragmented market in services, in particular. The IMF estimates that if you take all trade barriers on services inside the European Union and you add them up, it’s something like 110 percent tariffs. Trump Liberation Day tariffs, essentially, imposed within European Union. That means that European firms in digital and in services don’t have a harmonized market to scale into, the way the United States and China has. I think that’s by far the biggest reason.

On top of that, there are well-intentioned regulations like the GDPR that, by any account, has been a drag on innovation, and particularly been harmful for startups, whereas larger firms that find it easier to manage compliance costs have essentially managed to offset those costs by capturing a larger share of the market. I think the AI Act is going in the same direction there, ad so you have more hurdles, you have greater costs of innovating because of those regulatory barriers. And then the return to innovation is more capped by having a smaller, fragmented market.

I don’t think that culture or European lust for leisure rather than work is the key reason. I think there’s some of that, but if you look at the most dynamic places in Europe, it tends to be the Scandinavian countries and, being from Sweden myself, I can tell you that most people you will encounter there are not workaholics.

AI & labor (25:46)

I think AI at the moment has a real resilience problem. It’s very good that things where there’s a lot of precedent, it doesn’t do very well where precedence is thin.

As I finish up, let me ask you: Like a lot of economists who think about technology, you’ve thought about how AI will affect jobs — given what we’ve seen in the past few years, would it be your guess that, if we were to look at the labor force participation rates of the United States and other rich countries 10 years from now, that we will look at those employment numbers and think, “Wow, we can really see the impact of AI on those numbers”? Will it be extraordinarily evident, or would it be not as much?

Unless there’s very significant progress in AI, I don’t think so. I think AI at the moment has a real resilience problem. It’s very good that things where there’s a lot of precedent, it doesn’t do very well where precedence is thin. So in most activities where the world is changing, and the world is changing every day, you can’t really rely on AI to reliably do work for you.

An example of that, most people know of AlphaGo beating the world champion back in 2016. Few people will know that, back in 2023, human amateurs, using standard laptops, exposing the best Go programs to new positions that they would not have encountered in training, actually beat the best Go programs quite easily. So even in a domain where basically the problem is solved, where we already achieved super-human intelligence, you cannot really know how well these tools perform when circumstances change, and I think that that’s really a problem. So unless we solve that, I don’t think it’s going to have an impact that will mean that labor force participation is going to be significantly lower 10 years from now.

That said, I do think it’s going to have a very significant impact on white collar work, and people’s income and sense of status. I think of generative AI, in particular, as a tool that reduces barriers to entry in professional services. I often compare it to what happened with Uber and taxi services. With the arrival of GPS technology, knowing the name of every street in New York City was no longer a particularly valuable skill, and then with a platform matching supply and demand, anybody could essentially get into their car who has a driver’s license and top up their incomes on the side. As a result of that, incumbent drivers faced more competition, they took a pay cut of around 10 percent.

Obviously, a key difference with professional services is that they’re traded. So I think it’s very likely that, as generative AI reduces the productivity differential between people in, let’s say the US and the Philippines in financial modeling, in paralegal work, in accounting, in a host of professional services, more of those activities will shift abroad, and I think many knowledge workers that had envisioned prosperous careers may feel a sense of loss of status and income as a consequence, and I do think that’s quite significant.

On sale everywhere The Conservative Futurist: How To Create the Sci-Fi World We Were Promised

Faster, Please! is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit fasterplease.substack.com/subscribe