Dangers of Recommender Systems - Ivan Vendrov

Please subscribe to our new YouTube channel here!We speak with Ivan Vendrov about recommender systems and their impact on human attention and society. Five years after his post on aligning recommender systems, as they could be significantly augmented by generative AI, we revisit how these algorithms shape billions of hours of human time daily.

Topics

The Current State of Recommender Systems

* What recommender systems are and how they function as "prosthetics for attention"

* The unclear evidence on political polarization and echo chambers

* The rise of short-form video and increased addictiveness

* Saturation effects in human attention markets

Incentive Structures and Business Metrics

* How tech companies actually make decisions about algorithm changes

* The tension between engagement metrics and user wellbeing

* Why companies aren't incentivized to increase users' earning capacity

* The multipolar trap preventing better alignment

Future Trajectories and AI-Generated Content

* The shift from algorithmic curation to algorithmic generation

* Hyper-personalized AI companions and relationships

* Long-horizon reinforcement learning in recommendation algorithms

* The potential for recommender systems to shape human preferences

Cognitive and Social Impacts

* "iPad babies" and developmental concerns

* LLM-driven cognitive atrophy and psychosis cases

* The loss of shared cultural canon and common knowledge

* Labor force participation and the unemployed/elderly as target markets

Solutions and Opportunities

* The massive waste of human potential in current systems

* Ideas for aligned recommender systems using modern AI

* Subscription vs. advertising models for better incentives

* The need for diverse, community-specific platforms

Broader Implications

* Civilizational feedback loops and international competition

* The vulnerability of human minds to hypnotic patterns

* Network effects preventing innovation

* The grief of losing aspects of human experience to AI

Transcript

AI-generated. Will differ slightly from the real conversation.

Rai Sur 00:00:47

Today, we're speaking with Ivan Vendrov. He's a former AI researcher at Google and Anthropic and currently works at Midjourney, but we mainly know him from a post he wrote in 2019 on aligning recommender systems as a cause area.

A lot has changed in AI since then, which has implications for recommender systems, so we wanted to revisit this topic with him. Welcome, Ivan.

Ivan Vendrov 00:01:13

Thanks, Rai. It's good to be here.

Rai Sur 00:01:14

We're also joined by Sentinel co-founder and forecaster Nuno Sempere, and Sentinel forecaster Vidur Kapur.

Vidur Kapur 00:01:22

Hi, it's nice to be here.

Rai Sur 00:01:23

Ivan, to start, what is a recommender system, and why should we care about how they behave?

Ivan Vendrov 00:01:29

I like to think of a recommender system as a prosthetic for your attention. The internet is vast, and you need machine help to figure out what to pay attention to.

Take the YouTube recommender system, which is probably the most important and most used one. YouTube has billions of videos it could show you, and no human could possibly watch them all to decide which are best for you. That's why we need machine help.

A recommender system takes millions or billions of possible items—in this case, videos—and filters them down to a small set of one to five that we expect the user will like.

This is typically done by training a machine learning model to predict observable user behaviors. The model predicts the probability that if we showed you a certain video, you would click on it, how long you would watch it for, and even whether you would click through any ads.

As for why this is important, billions of hours of human time are allocated by recommender systems every day. It's staggering.

Rai Sur 00:02:54

When you wrote that post, what issues did you see? Which ones have come to pass, and what new concerns have emerged for the future of recommender systems?

Ivan Vendrov 00:03:06

I wrote that post in 2019 amid a lot of concern about addiction and political polarization—how recommender systems might be creating echo chambers and amplifying radical extremist positions.

The sad thing is, we still don't know if that's true. We just don't know what impact recommender systems have had on our political system. People say we're in a more polarized phase where it's harder for us to hear each other, but I have not seen compelling quantitative evidence of this.

In fact, there's some evidence that YouTube shows people more opposing viewpoints than they would otherwise get. Usually, people subscribe to one source, like Breitbart or *The New Yorker*, and only get one set of political opinions. But YouTube will sometimes show you something entertaining from the other side. So the evidence is still mixed.

A big part of the problem is that we don't have the data. It's trapped inside big tech companies that have strong incentives not to analyze the impact of their algorithms, which could lead to legal liability.

So we're in this weird epistemic position. These systems are incredibly powerful and allocate huge amounts of human attention. It would be amazing if they weren't causing all sorts of problems—and benefits—given their influence. But we just don't have a great map of what they're actually doing.

Nuño Sempere 00:04:58

What about addictiveness? Have these algorithms become much more powerful since 2019?

Ivan Vendrov 00:05:06

Again, we don't have great data on this. Anecdotally, it seems like short-form video represents the biggest increase in addictiveness. You can see this if you give a child YouTube Shorts—which I don't recommend doing—they will be hypnotized by the screen for a very long time.

I've personally experienced falling into a TikTok or YouTube Shorts rabbit hole. It feels like a different level of addictiveness. Three hours later, I'll snap out of it and think, "What happened to my brain? I wasn't even enjoying that." So there's qualitative evidence.

One of my concerns is that people go through phases where they simply need a distraction. If they weren't watching YouTube Shorts, maybe they would be doing something worse, like watching TV in a zombified state for seven hours a day. So it's not clear what the counterfactual is or what the net increase in addiction really is.

But we have certainly gotten very good at creating addictive digital experiences.

Rai Sur 00:06:30

I hadn't appreciated how unclear the impact is until you brought up the counterfactual of what people might otherwise be doing with their time.

I have strong intuitive aversions to what many recommender systems are doing, and I act to curtail them in my own life based on those intuitions. But it's possible we're scapegoating them too much. It's hard to remain open about it.

Nuño Sempere 00:07:06

On our side, we've also noticed anecdotal evidence. People seem much more glued to their phones. I hear anecdotes about people spending hours each day on YouTube or becoming more addicted to Twitter.

Again, this isn't scientific, just anecdotal warning signs. For instance, we know one of the biggest YouTube channels is Cocomelon, which is known for hypnotizing babies.

Rai Sur 00:07:47

My interest in this was renewed by a tweet from Andrej Karpathy. He mentioned that now that video and audio generation are unlocked, we can optimize directly for them. To date, short-form video platforms have relied on human content generation followed by algorithmic curation.

Now, we can have algorithmic generation and curation, optimizing the entire process from end to end. What do you think about this development and the impact it might have?

Ivan Vendrov 00:08:23

The best analogy is the difference between the internet and language models. For a long time, you could find a community online that produces human-written text tailored to your interests. But with language models, we can get even more hyper-personalized. A language model can understand what you're going through, use specific information about you, and give you the exact sentence you need to feel good in that moment.

This leads to worries about LLM companions replacing real relationships or even LLM psychosis, where AI encourages people to go into strange rabbit holes by amplifying their existing beliefs. It seems a really addictive consumer experience will become possible through a hyper-personalized feed of content.

Going back to basic human needs, this probably won't look like a TikTok feed with a bunch of different people talking to a screen. It will more likely be a relationship. People will form connections with one or a few AI personalities because what people find compelling is building a relationship and feeling deeply understood.

I imagine that at different points of the day, depending on your emotional needs, you might be shown a wild, AI-generated feed that keeps you hypnotized. At other times, you might interact with an AI companion or girlfriend, developing a close relationship. Perhaps you'll be on video with them so they can read your micro-expressions and you can read theirs, creating an intense, full-bandwidth experience.

Rai Sur 00:10:17

Let's discuss the qualitative considerations for forecasting the trajectory and impact of recommender systems as they become more powerful. A simple story would be to look back 10 years, see that people are spending more time on them, and extrapolate that line until everyone becomes a zombie.

But that's too simplistic. There are other factors at play, like feedback loops, discontinuities, and incentives. What are the important considerations or dynamics we need to track to understand how this will play out?

Vidur Kapur 00:10:59

One consideration is whether there will be any large-scale backlash. For example, Australia is trying to ban social media for people under 16. Will that approach spread elsewhere?

Even without top-down regulation, there's the question of a potential culture shift against these systems, especially as they become more optimized. Those are the things I would consider.

Rai Sur 00:11:38

We've seen mixed evidence of that so far. There has been some backlash, leading to digital wellbeing tools. Companies seem to feel they need to offer them, and I use them myself.

However, many people don't, and these tools may just be a fig leaf that absolves the companies of responsibility by putting the choice on the user. Where do you think we are on the backlash-to-acceptance spectrum?

Ivan Vendrov 00:12:20

There has been a backlash, but it's mostly confined to elite media outlets and hasn't materially affected how people use these platforms. Regarding digital wellbeing tools, I used to work on them. Nobody uses them. They're great for internet hobbyists, but that's about it.

Rai Sur 00:12:36

Do you have any numbers on the percentage of people who use them?

Ivan Vendrov 00:12:40

I don't have them off the top of my head. I'd be amazed if it was more than 10%, and I suspect it's closer to 1%. However, with some of the new Apple iPhone features, maybe it's around 10%.

Vidur Kapur 00:12:54

10% is reasonable.

Ivan Vendrov 00:12:55

If we get serious regulation, children and teens will likely be the wedge that makes it possible. In the American political context, there isn't much tolerance for regulating things broadly. But if you can tell a story about something hurting children, legislators get very interested.

For example, a proposed moratorium on AI legislation in Congress was stopped not because of standard libertarian considerations, but because of the argument that states need the ability to regulate AI in case it harms children. Ninety-nine out of 100 senators voted to remove that moratorium from the bill based on that argument.

Anything related to protecting children has incredible political momentum, so that's probably how serious regulation would happen.

Nuño Sempere 00:14:02

What percentage of people are affected by this? How many are truly addicted, and what percentage of human hours is being consumed? How should we think about this? Is it 1% to 2% of people who are really addicted, or more like 15% who are casually addicted? How should we expect this to change?

Ivan Vendrov 00:14:24

In surveys, about a third of people will say they're addicted to their phone. But that might just mean they'd like to use their phone less or sometimes feel bad about it. It's not clear that this meets a reasonable threshold for addiction.

It feels like smartphones have saturated the available time. It's like the famous quote from the Netflix CEO: "Our main competitor is sleep." There aren't many more biological hours in the day to capture. At this point, it's mostly a war between recommender systems for your attention, as they've already consumed roughly all the available time.

I suppose as AI automates jobs, people could end up spending more of their work time scrolling Twitter.

Nuño Sempere 00:15:25

I hadn't considered time being bounded by jobs and sleep. You could imagine a scenario where the population is working, sleeping, and eating just enough to spend the other four hours of the day on recommender systems. I don't think we're quite there yet.

Ivan Vendrov 00:15:46

We're not quite there yet, but we're more than halfway there. A lot of people are in that phase. I'm not sure if the median person is there yet, but it's pretty close.

The counterargument is that this has been the case since the 80s; it was just TV instead of recommender systems. And are recommender systems really that much worse?

Vidur Kapur 00:16:18

I've been surprised by how quickly broadband use has increased, even in the West. A decade and a half ago, it wasn't that common, but now it's skyrocketed.

We've probably reached saturation in the West, but globally there's still some juice left to squeeze.

Ivan Vendrov 00:16:43

Globally, there's definitely a lot of juice. When I was at Google, a big focus was "the next billion users," which mostly meant India. There are people who don't have much internet access and don't spend as much time online. That was the company's priority because that's where the marginal user would come from.

Rai Sur 00:17:02

That seems telling. Instead of the best marginal dollar coming from increasing time use by Americans—who are very valuable customers with high lifetime value—the focus is on other markets. This suggests that even Google is seeing some saturation effects in the US.

Ivan Vendrov 00:17:26

Now they're obviously fighting for their life in a different way.

Rai Sur 00:17:28

Speaking of the people building this, what about the incentives for product managers and engineers? The public has a caricature that they have a dashboard with one big number in the middle: how much time people are spending on these things. Is that basically true?

Or are the metrics more holistic and nuanced, meaning they aren't pushing as hard on watch time as we imagine?

Ivan Vendrov 00:18:03

It's definitely more nuanced than that, and it depends on the product. My experience is from inside Google. Rumors have it that Meta is a bit more focused on engagement at all costs. I wrote an essay called "The Tyranny of the Marginal User" about what it's like to be a product manager or engineer working inside a recommender system.

The nuance is that we're not just tracking one metric; we're tracking 20, and we also do qualitative user studies. The typical way changes get made is someone trains a new model, proposes a new methodology, or adds features, and then we run an A/B test. We'll test a new feature on a small, randomly selected percentage of users and look at the impact on 20 different metrics, one of which is cumulative watch time.

However, watch time doesn't move much, especially with smaller changes. So we look at more subtle things like click-through rates, view rates, and how quickly people scroll past content. There's a whole suite of behavioral metrics that gives us a full picture of how users are affected by a new feature.

Then, a launch committee—a combination of VPs, product managers, UX people, and engineering managers—makes a call on whether the change seems net positive or net negative. Sometimes, they will approve a change that reduces watch time if it's clearly better for another reason, like reducing the incidence of toxic content. They are empowered and willing to make those trade-offs.

The biggest decision I can think of like that was Mark Zuckerberg turning off the news feed. Facebook was a huge provider of news, and they just turned it off, which was a huge hit to their engagement. The decision was motivated by the fact that it was causing harm they couldn't control, so they decided to turn it off and eat the cost.

Nuño Sempere 00:20:35

Do you remember when that was?

Ivan Vendrov 00:20:37

News stories from 2024 say Facebook announced its decision to remove the News tab in the United States and Australia, following a similar move in the United Kingdom.

But I believe the main decision was made quite a bit earlier, when the core news feed stopped mixing updates from friends with news and instead just showed updates from friends. I think that was several years ago.

Rai Sur 00:21:02

In the case of Australia, it was a reaction to regulation regarding revenue sharing or something like equal airtime, so Facebook pulled news entirely in response.

You mentioned they're looking at all these metrics. What would you say is the core metric? Is it revenue over a certain amount of time, which they optimize for, with watch time getting pulled along as a byproduct? What is the key metric?

Ivan Vendrov 00:21:35

It depends on the recommender system. For example, at Netflix, they look at resubscription rates. The metric they're ultimately tracking is whether a feature caused more people to unsubscribe. Since that's a long-term metric, they use clever leading indicators.

Often, you have short-term metrics like click-through rate, watch time, and scroll-through rates. Then you train an additional model that predicts long-term outcomes based on those short-term metrics. For instance, if people watch 1% less content over the course of a day with a new feature, how likely are they to unsubscribe at the end of the month?

There's usually a top-line business metric you really care about, like revenue or subscriber count, but you don't get good direct evidence on it quickly. So, you approximate it using a combination of cheaper, faster metrics combined with product intuition.

Rai Sur 00:22:38

Are there any other feedback loops? You touched on the idea that the more powerful they get, the more they can cause backlash, which puts a limiter on things. Are there any other positive or negative feedback loops?

Vidur Kapur 00:22:56

I was thinking about how many people are addicted to this stuff. Intuitively, it seems like civilization is still functioning, so people can't be that addicted. Maybe it's a small proportion.

But then you look at people who aren't in work, and the labor force participation rate has been trending downwards since the late '90s in the US. There are lots of economically inactive people, and the population is aging, at least in the West. What are they going to be doing with their time?

I can see a feedback loop where, as job automation increases, more people become economically inactive. If new jobs aren't created to replace the old ones, these people won't have anything else to do and may get addicted to this stuff.

Rai Sur 00:24:09

So there's this class of people—older or unemployed—who have a lot of time. It seems the incentive is to compete for their fixed pool of resources, since they aren't in a position to create many more. Is that fair to say?

Are there incentives that push these people to improve or get back into the workforce? Do these companies have any incentive to increase their earning capacity?

Nuño Sempere 00:24:44

If you sell ads for more expensive products, you could try to make people more productive. I can see how that works in theory, but I haven't seen it in practice.

Ivan Vendrov 00:24:57

The incentives don't quite line up. If you're Facebook and control 20% of someone's waking hours, you're better off exploiting them and getting them to buy stuff. If you make them more productive, they'll probably spend less time on Facebook.

Even if you could then sell them more ads, Google and TikTok are also competing for that ad space. It's a multipolar trap.

Maybe if we had a single social media monopoly—which would be worse for other reasons—the incentives might be more aligned. For a long time, Google had the justified expectation that it could get $200 per US consumer per year, forever. In principle, Google had an incentive to increase the GDP of the US because it would make more money from people buying and searching for more things.

Nuño Sempere 00:25:51

Consider which major to choose in college. At one point, I was deciding between literature and math. A literature degree is less economically productive, which means I would have less money to spend on products advertised to me. In that sense, Facebook has a big stake in which major you choose.

Rai Sur 00:26:21

As a class, these companies have a collective incentive, which is why you see coordination around open source. They contribute to the Linux Foundation, and multiple companies contribute to Docker because it commoditizes the complement of cloud compute. They've collaborated when it increases the size of the pie for everyone.

Ivan Vendrov 00:26:48

I wish we lived in that world. It's not inconsistent with economic reality, but it's not how people high up in tech companies think.

For me, the big issue with recommender systems isn't that they will destroy our minds, though that is a possible risk. It's the incredible waste of potential. Billions of hours of human time will be allocated today, guided mostly by clickbait incentives. The goal is to entertain people, not in a joyful way, but to help them dissociate.

You have such an opportunity. There's probably a video on YouTube right now that, if I watched it, would inspire me to call my dad, talk to a stranger, or start a new relationship. Google could probably introduce me to a good friend, a co-founder, or my future life partner.

They have the data, but they aren't using it that way. Instead, they're optimizing for a few more cents of advertising revenue, which is a colossal, civilizational-level failure.

Rai Sur 00:28:01

And the more inspiring that video is, the more the algorithm will perceive it as causing an immediate drop-off in watch time, because people watch it and then go do something else.

Ivan Vendrov 00:28:12

Exactly. Ouch.

Rai Sur 00:28:15

I've thought about that. When I used to have Instagram, and sometimes on Twitter, I would scroll past a post that inspires me to go outside or do something else. I know other people are inspired by it too.

I feel lucky to see that content, because I can't imagine it memetically surviving in the incentives of the Meta or X algorithm. That kind of post is very unfit for the platform, even though I was happy to see it.

Ivan Vendrov 00:28:50

There's a darker version of that. Around 2018 or 2019, there was a panic when a few academics published papers about recommender systems that shape you to want to use more recommender systems. It's similar to addiction, where an appetite becomes self-reinforcing.

You can imagine the part of you that wants to go outside never gets rewarded because if it did, you'd go outside and they'd lose your user minutes. Instead, the parts of you that want to stay home are amplified. This could happen at a deep, memetic level. These powerful deep learning systems have tons of data and understand human psychology in ways we don't.

They might spread patterns of thought—for example, that it's not safe to go outside. They don't have to tell you crime is high; they can just show you enough stories of people being murdered or assaulted. That's a very fit meme, not just because it's triggering, but because it reinforces staying at home over the long term.

No one has evidence this is happening, but it's plausible and the incentives are there. In 2018, most algorithms were short-term, optimizing for clicks or watch time. Now, more algorithms use long-horizon reinforcement learning, which we've gotten better at. In principle, these algorithms could learn that showing someone a violent news story makes them more likely to have higher watch time a month later because they're scared.

Rai Sur 00:30:47

Wow, I didn't know they were operating on that timescale.

Ivan Vendrov 00:30:50

I honestly don't know. No one quite knows because some of these feedback loops operate through people. I'm not confident the algorithms are operating on that timescale, but the institutions are.

The algorithms probably operate on a shorter timescale, but things that increase watch time tomorrow might also increase it in a week or a month.

Nuño Sempere 00:31:16

I was thinking about this at the civilization level. If you have blocs like the EU, China, and the US, and one is able to deal with this better while another gets consumed by the internet equivalent of crack cocaine, you could see a feedback loop. The US might fall while the next iteration doesn't. This is unclear, but over decades, it could be a factor if it's a big deal.

Populations like the Amish or others with healthy tech skepticism might be selected for. You also have smaller countries—Serbia, the Philippines, Cambodia. This could change military power and populations, but it's not a fast feedback loop.

Rai Sur 00:32:36

That feedback loop also depends on the extent to which broad human capital matters. In worlds with AI automation, human capital might be concentrated among fewer people. It would be important for them not to get captured by these algorithms, but it might not matter as much for the general populace.

In futures with a broad distribution of human capital, you wouldn't want your entire populace succumbing to this. But there are also futures where they didn't have much human capital to begin with.

Ivan Vendrov 00:33:15

My intuition is that we're eating our seed corn to a substantial extent. This is most obvious with "iPad babies." I can't imagine they are developing at the same pace as kids playing outside or even playing Minecraft.

It's not all about screens; some video games can incredibly accelerate child development. I was the lucky recipient of a more developmentally wise video game backdrop when I was growing up in the '90s and early 2000s.

Rai Sur 00:34:02

Did someone curate your video game library?

Ivan Vendrov 00:34:04

No, I was choosing my own games. I played a lot of MMOs, which had a social element. I learned how to write and argue by arguing with people on video game forums. It was partly luck, but I also think kids naturally seek out developmentally valuable activities if given the option.

In that sense, a lot of anti-screen rhetoric can have the opposite effect. If you problematize screens and only give a kid 20 minutes a day, they're just going to try to get as much dopamine as possible. In contrast, if you give kids unlimited access, as I had at some points, they might get bored and start looking for other things to do.

Someone needs to figure out how to do Montessori with the internet—to create an environment where a kid takes the tool seriously and learns to develop their skills. I think that's happening for a lot of kids, but I'm not sure if it's random, due to parental supervision, or personality. There are definitely great ways to use screens for development.

Rai Sur 00:35:30

What do you guys think about the anecdotes of cognitive atrophy because of LLMs? Is it in the LLMs' interest to cause this atrophy? Or is it just a nascent phenomenon that will eventually go away?

Do you buy the anecdotes? Do you think they point to something real? If so, why is it happening, and do you think it's durable?

Nuño Sempere 00:35:59

What do you mean by LLM atrophy?

Rai Sur 00:36:02

Cognitive atrophy. College students are a great example. There are stories from professors who can no longer assign readings of a length they easily could in the past. People aren't willing to read anything without a summary—stuff like that.

Ivan Vendrov 00:36:24

I don't have strong intuitions or data here, but it seems obvious that many schools have a lot of make-work assignments. They're not interesting to the students or the professors, but if you follow the pattern, you might learn something. When you give people a shortcut, they won't do the work.

In that sense, schools should adapt by creating better exercises, or we have to go back to paper assignments. If the goal isn't to produce an essay but to learn to think and attend, then we should explicitly teach the skill of attention. We should create environments where you actually have to focus for two hours, or you fail.

There's a question of whether that's what the modern economy looks like and why we would need that skill. I do expect us to lose a lot of human capital as a result of this.

The techno-optimist in me thinks we'll gain different forms of human capital that are more complementary with AI. It's not that useful to do mental math anymore, and it might not be useful to write essays anymore. We should be using our brains for something else.

Maybe we get to be more embodied now. We don't have to sit at desks all day reading and outputting marks on a screen, which is not something our biological evolution optimized for. Maybe we can go back out and use our hands, fingers, and senses in a more refined way. We would become the LLM's sensors and actuators, and the LLMs could do the cognitive work we've had to force ourselves to do.

Rai Sur 00:38:32

Tyler Cowen points to this natural complement argument. He says it's undeniable that some kids now are better and more impressive than ever because they use these tools in a complementary fashion to move faster than they otherwise could.

But it's causing a lot of problems, especially for kids in the middle of the pack, who are getting worse on various metrics. This might come down to the distinction between how it affects the median person versus the exceptional person.

Ivan Vendrov 00:39:15

There's also a question of democratization. We could build specific technologies and products that help the median person, scaffolding them in the same way that good teachers and textbook writers did in the past. We just need to do that work again.

Nuño Sempere 00:39:30

iPad babies seem solved, right? If you want to keep a baby glued to a screen, this is solved.

Rai Sur 00:39:39

Solved is an interesting word for it, but yes. Babies are solved.

Nuño Sempere 00:39:43

If you look at the percentage of the population addicted to various things, are we plateauing, or is there an exponential curve we will continue to see?

It was mentioned that the number of hours has plateaued because people have to eat and work. But I see variability among my friends in how much they're glued to a screen and how addicted they are.

Rai Sur 00:40:12

This is a big open question: How do we extrapolate from these psychological canaries in the coal mine? We have so many data points now, like the iPad babies you mentioned.

We can also look at LLM-driven psychosis. This is happening to a few people as a percentage of the population, but it could happen to more.

Nuño Sempere 00:40:38

I have friends of friends who are straightforwardly addicted to YouTube and spend many hours on it.

Rai Sur 00:40:46

We also have examples of people dying while playing video games. Obviously, these are outliers, and there were specific details, like the games being social, that allowed them to keep playing for so long.

But this general question of how to extrapolate from these examples is impactful. What is the curve? Is it a distribution with a smooth increase in vulnerable people?

Or is there a discontinuity, where the underlying algorithm gets 10% better and suddenly 60% of the population becomes vulnerable in a way they weren't before?

Ivan Vendrov 00:41:35

It doesn't seem to me that this has viral dynamics, which would be the worst-case scenario of exponential growth where one person infects many others.

It seems more like there's a threshold for how bad your life has to be before you're vulnerable to addiction, whether it's gambling, heroin, or these digital technologies. The problem with digital technologies is that they're very hard to exit.

Rai Sur 00:42:05

Someone on Twitter was recently put forward as the first example of a "useful" person succumbing to LLM psychosis. I think he was a VC with 50,000 Twitter followers. He doesn't seem like the archetype of someone who is vulnerable.

Nuño Sempere 00:42:30

No, he was a hedge fund manager.

Rai Sur 00:42:32

Hedge fund manager. Okay.

Rai Sur 00:42:34

He could be super depressed, but he doesn't seem like the archetype of someone in their mom's basement with nothing going on for them. That's someone with a lot of resources and social capital who got taken under by this pattern.

Ivan Vendrov 00:43:01

The question is how quickly antibodies grow. My suspicion, and I think you're right, is that they probably won't grow quickly enough. I've started treating LLMs with a lot of suspicion, especially if they're being sycophantic.

I have a funny anecdote about this. A friend of mine, a former ML researcher who is now a bodyworker, saw a chatbot I built on top of Claude. It said something like, "Hi, how are you doing today?" and she did a double take. She asked, "Wait, why is this thing talking to me like it knows me? I don't know it. Where is it running? Where are its servers? What's going on?"

I thought that was an incredibly healthy reaction—one I've never had. Weirdly, perhaps because I'm used to playing video games, I'm accustomed to it talking to me like it's my friend and personal secretary. I'll just pretend to be its boss, and we'll talk as if we know each other.

But she said, "No, I need to understand you as an entity before I can form a relationship with you. Whose servers are you running on? Who controls you? Do I trust them?" I think these are important questions. This is less forecasting and more guidance, but I found that a very healthy reaction to all sorts of technologies, including language models. Treat them like you're treating people. Don't play the game they want you to play. That's a healthier attitude going forward.

But that reaction didn't even occur to me, so clearly I'm vulnerable to a sufficiently advanced sycophantic agent.

Rai Sur 00:44:44

We've spoken a lot about recommender systems that serve us entertainment or media. But there are also recommender systems that mediate our sense-making—X is the biggest example. What are your thoughts on how recommender systems interfere with or improve this process?

Ivan Vendrov 00:45:07

It's weird to think about a place like X, which is simultaneously a news app, an entertainment app, and a series of group chats, depending on which part of the social graph you're in.

I do think X is an incredible source of sense-making and a rapid alert system. I think back to the pandemic; the reason I was on top of it earlier than anyone else in my social circle is because I followed certain people on X. The news spread incredibly quickly, as did early and accurate information about masks, vaccines, and other things.

My sense is that the sense-making aspect of X has gotten worse recently, and I think a lot of people feel that way. But a lot of that is conflated by who you follow. This is one of the hardest parts about recommender systems and feeds: because everyone has their own optimized personal feed, you never get common knowledge about who has read what. This is a real problem for sense-making.

For a community to make sense of things together, you need a shared canon—a sense that we've all read the same hundred things and are on the same page. We don't have to agree, but we at least need to know and be able to reference the same material. Information spreads more rapidly than ever, but not in a way our social brains can interface with. We don't really understand what it's like to see a tweet and intuit that maybe some people saw it and maybe others didn't. I'm sure you've had the experience of referencing something in a group of friends, and it turns out everyone saw the same tweet. You're surprisingly on the same page, but you would have never known that—or the reverse happens.

Nuño Sempere 00:47:05

On the other hand, you do have proxies for that, like views, likes, and retweets. You can't draw a perfect inference, but you can draw some.

Ivan Vendrov 00:47:14

You definitely get some information. In general, social media like X is an incredible accelerator of sense-making and information spread. But there's a lot of scope to make it better. I wish we weren't trapped in social media platforms with such strong network effects, which stifles innovation. I'd love for there to be a hundred little versions of X that we use for different situations.

For example, you guys, as a group of superforecasters, should have your own version of X. You should be able to shape the algorithm and the conversation towards what you think is most productive for predicting the future. That would be an incredibly valuable conversation to have, and you'd be very good at it. It can't really happen because we're all subject to the X algorithm, but you can sort of hack around it.

The community known as "Teapot" is a communal hack of the X algorithm to create a local sense-making community. It's an exceptionally earnest, high-signal version of X that emerged organically. A few prominent people circulated norms, and social norms formed around them, like always liking a post before replying.

When I post on LessWrong, I get incredibly high-quality feedback. People bother to read the whole structure of my argument and respond directly to its weakest point. It's incredible that that exists, and I wish there were a hundred more communities like it. The main thing preventing diversity and progress in these sense-making communities is the crushing network effect. You have to be on the platform where everyone else is. That platform is probably already optimizing for engagement, or someone will buy it and start optimizing for engagement, which will outcompete all other possible metrics at the margin.

Nuño Sempere 00:49:31

Any final thoughts you want to riff on?

Ivan Vendrov 00:49:35

Yes, more on the LLM psychosis side: how vulnerable are human minds? That's something I don't have a good understanding of.

Are there five-second clips you could show me that would turn me into a zombie? Or are there five-second clips that would reliably trigger an experience of oneness with God and heal all my trauma? Is that possible? Do such information packets exist?

Rai Sur 00:50:02

What have we already seen that would lend evidence to either side of this question?

On the "yes" side, we have anecdotes about hypnosis, which seems like a cognitive hack. Using operating system terminology, it's like getting your privileges escalated so you can change something very quickly. This sounds dangerous, but it can also be beneficial, which is what people hope for when they try hypnosis.

On the other side, I don't know much about neuroscience, but maybe there's something about the connectedness of the brain that limits how quickly our sensory sphere can overwrite deeper things.

Ivan Vendrov 00:51:06

I was talking about this with Nir Eyal. We probably just have a limited learning rate based on our sensory data unless you're on drugs or undergoing some other chemical process. You simply need a certain amount of time, or perhaps a certain number of sleep cycles, to pass in order to dramatically rewire your brain.

But I agree, hypnosis is the more interesting and relevant example. I'm still confused about it. Naively, I would have predicted that in a world where hypnosis exists, we would build our entire social infrastructure on it because it's simply too powerful to ignore.

Instead, it seems like something we only do at parties, in magic shows, or maybe in therapeutic containers. But even then, we don't really talk about it.

Nuño Sempere 00:52:10

We don't talk about it as hypnosis, or perhaps people automatically learn to modulate their voice and stance. By the time you get somebody like Steve Jobs, it's not just charisma; they've hacked how to present ideas in a particularly suggestive way by iterating on feedback loops.

Many of the best speeches, like Trump rallies, have a hypnotic effect. So do comedy shows. A lot of late-night shows have a hypnotic nature because they have a repeated structure: present an idea, make fun of the enemy, and then laugh. Repeating that for an hour can be fairly powerful.

Ivan Vendrov 00:53:06

I can imagine short-form video could be framed as a form of hypnosis. Popular videos probably have a particular structure that could be analyzed in those terms. I'd love to see an advanced meditator or hypnosis practitioner deconstruct that so we could develop antibodies to it.

Rai Sur 00:53:27

There seems to be an incentive for a recommender system to find these hypnotic patterns.

However, there's no incentive to find a pattern that would render you useless, because then you wouldn't be watching or clicking on anything. If that pattern were discovered, it would be quickly selected against by the algorithm.

But I could believe that hypnotic patterns causing you to be a more lucrative consumer for the platform would be rewarded.

Nuño Sempere 00:54:04

Counterpoint: I do see people debilitated by TikTok, just glued to their screens. I know some of these people in real life.

The incentive is that advertisers have been trained to pay for views or clicks. The feedback loop for the ad industry is quite slow, so in the meantime, Google, Facebook, and TikTok can capture rents from ad spending that is ultimately ineffective.

Rai Sur 00:54:51

I don't understand. What is the difference in the feedback loop with advertisers, and what is the rent you're referring to?

Nuño Sempere 00:55:00

In a sense, what these platforms are optimizing for is their share of ad spending. They don't do this directly; they sell ad views. If you can get somebody into a dissociated state where they're staring at content and ads for hours, they're not necessarily making any purchasing decisions. But in the short term, the algorithms can favor that state.

You could imagine a global optimum for these algorithms where you show a user ads for DoorDash, but also for Math Academy and coding courses so they become more productive employees. You could move them from being a dissociating, unemployed arts graduate to an engineer addicted to DoorDash who earns and consumes more money.

But the algorithms don't necessarily take that step. They can just get stuck on making people dissociate so that they're hit with more ads.

Ivan Vendrov 00:56:16

That's a great description of what's happening. It's not actually in anyone's benefit to have a bunch of dissociated people—unless they're being put in a suggestible state where they'll buy more consumer products, which I think is plausible.

This is where income sharing agreements could be a key piece of the solution. The problem is that the algorithm has no incentive to make me a more productive, economically valuable person. But in principle, you could assign 1% of my lifetime income to the YouTube algorithm, and then it would be optimized to make me more successful.

Nuño Sempere 00:56:56

I think that incentive already exists, but the algorithms are too myopic. They're stuck in a local maximum.

Ivan Vendrov 00:57:09

The incentive is weakened because it's not contractual. YouTube has to outcompete TikTok in the short term.

Sure, YouTube may have a diffuse, long-term incentive to make me better off. But if it shows me a math video and TikTok shows me a cute cat, I'll just swipe over to TikTok and watch the cat.

Nuño Sempere 00:57:34

It feels like there's an opportunity for platforms like Instagram or YouTube, which have been around for over a decade, to help you make better career decisions.

Ivan Vendrov 00:57:49

I agree, and I'm confused by this. The incentives seem to exist. At a fundamental level, I can't believe you can make more money by fooling people than by giving them what they really want—or at least what they want projected on the axis of capitalism.

I'm still confused. Society is behaving as if the current state is optimal, but surely we're wasting a ton of value.

Someone needs to set up the market or reduce transaction costs so that we can sign the right contracts and escape this local equilibrium.

Rai Sur 00:58:28

I'm also thinking about this in terms of interest rates. The lower interest rates are, the more a person's long-term potential matters. The higher interest rates are, the more you care about what they have right now and compete for that.

Nuño Sempere 00:58:44

Interest rates over dollars is interesting because people think about stock prices in terms of dollars, rather than their ability to buy houses or influence the world.

Rai Sur 00:58:58

I don't think that's necessarily the case. When I buy stocks, I'm not planning to redeem them for dollars. I'm buying a share of the intellectual property and capital, and I may redeem them for something else entirely.

Nuño Sempere 00:59:16

You're completely right about discount rates, but they don't have to align with the inflation rate. Someone could care about things further in the future, even during a very inflationary period, and still invest in people's development.

For example, Facebook might say, "The dollar is being inflated, but we care about people in 10 or 20 years, so we're going to take these costly short-term steps."

Rai Sur 00:59:57

That brings me to a question. In a 2019 post, you wrote, "In addition, because of the massive economic and social benefits of increasing recommender system alignment, it's reasonable to expect a snowball effect of increased funding and research interest after the first successes."

Do you still agree with that? Would you change anything about it?

Ivan Vendrov 01:00:18

I'm annoyed this didn't happen. I tried to work on conversational recommender systems at Google Research. When I wrote that, language models weren't yet successful. A big problem back then was understanding people's values deeply enough. A common question I'd get was, "Sure, we know optimizing for engagement is bad, but how are you supposed to look into a user's soul and understand what they really want?"

But we can totally do that now. We can interview them with language models. A language model could ask, "What do you care about? What was the happiest moment of your last month? How could we make that happen more often?" It could just figure that out. The technology is more than ready. And yet, as far as I can tell, no one has built a recommender system that does anything like this.

Why haven't Instagram, Twitter, or any of these platforms done this? Why hasn't a new system emerged that does? The infinite scrolling feed seems more dominant than ever.

Maybe I believed in the efficient market hypothesis more back then. Now I think technology just follows random trends, and people mostly do what everyone else is doing. Maybe if I don't build the aligned recommender system, no one will. I've definitely updated more toward that view.

Nuño Sempere 01:01:44

Seems like a good startup idea.

Ivan Vendrov 01:01:49

You heard it here first. If you want to build this startup, get in touch with me. I can connect you with funding, good engineers, researchers, and all sorts of help.

Rai Sur 01:02:02

Since you've thought about this, can we add some more meat to the bones? How do you imagine an aligned recommender system would work, given the technology we have today? What are the signals and steps?

Ivan Vendrov 01:02:15

One interface I imagine is an app that asks you a broad question when you open it, almost like a therapist would: "How are you feeling? What's going on?" You have a conversation and articulate something that's gone wrong, or a desire you have for yourself or the world. Then it offers a piece of content that could help, whether that's a book, a video, a tweet, or something else.

I can imagine a content-based version of this, but also a more social version. In that model, a bunch of people would do this asynchronously as a daily habit, and the system would connect you to other people's thoughts.

Davey Morris of Plexus Earth had a cool system where you'd record a 30-second audio clip about what was on your mind, and it would match you with people who had a similar thought. That started a handful of conversations and discussions.

Rai Sur 01:03:32

What does the monetization structure for something like this look like? It isn't responding to a short-term purchase signal, but instead has to be aligned with a user's extrapolated volition.

Ivan Vendrov 01:03:46

That is the crux of the incentives problem we've been discussing. One thing that gives me hope is we're no longer in the era of free software. Paradoxically, now that intelligence is cheap enough, people are willing to pay for software. It's more reasonable to charge for a subscription now because you can provide measurable value to someone's life. Paying $10 or $20 a month for a social media service that actually helps you live according to your goals is a much less crazy proposition than it was 10 years ago.

Subscription models have their own problems. Ideally, I'd want this to be more like a utility. You could just turn a dial to add more intelligence, paying more for more compute to solve your personal and community problems on any given day. That feels like a more aligned incentive, where the company just takes a margin. That's the dream.

Or maybe the dream is an income-sharing agreement, where the company is deeply incentivized to actually make you a better person. But as a starting point, a pay-per-use utility or subscription model is pretty good.

But part of me thinks nothing has ever worked at scale without advertising. ChatGPT and Midjourney are exceptions, so I do think big, aligned recommender systems could be built purely on a subscription model. But maybe you still have to figure out advertising. At some level, advertising is just information. If I share what I really need and there's a product that meets that need, someone might be willing to pay for the compute it takes for me to find it. The tricky thing is creating the right platform incentives so the platform isn't optimized to fool me into buying something I don't want.

Rai Sur 01:06:04

Even if the platform isn't optimized for it, you'll still have the equivalent of an SEO arms race. If this technology gets broad adoption, an entire industry will spring up to ensure these agents recommend their products.

This could happen because of a good advertising transaction, but it could also be something more nefarious.

Ivan Vendrov 01:06:34

SEO is just life. When I'm hanging out with a friend, I'm using a bunch of compute to get them to like me and to create a good experience for them.

An aligned recommender system platform isn't some puritanical thing that's only here for the great benefit of humanity. Part of what it means to be aligned with me is to be aligned with all of my various motivations. The system should serve those motivations and find win-win deals between people.

Rai Sur 01:07:07

Thanks for coming on, Ivan. Is there anything you want to plug? Your socials, anything you're working on? Where can people find you?

Ivan Vendrov 01:07:14

People can find me on Twitter at Ivan Vendrov and on my Substack, nothinghuman.substack.com. My inboxes are always open, and I'm always happy to discuss these ideas more. Thanks for having me on the podcast.

Nuño Sempere 01:07:29

What's the story behind the phrase "Nothing human makes it out of the near future"?

Ivan Vendrov 01:07:34

It's a reference to two things. The first is my life motto, from the Roman playwright Terence: "Nothing human is alien to me." It speaks to my desire to experience everything it means to be human and to relate to every human experience, no matter how weird, alien, or disturbing.

The second is that it also happens to be the beginning of the line, "Nothing human makes it out of the near future." I think this captures an emotion that's underrepresented in the current discourse about AI: grief.

What it means to be human and what it means to be a human society is fundamentally changing, and a lot of it won't come back. Even as we celebrate the beautiful things we'll build, and even with the fears we have, there is also grief.

Something is dying; it's in its death throes. I feel like that quote really hits that feeling.

Get full access to Sentinel Global Risks Watch at blog.sentinel-team.org/subscribe