Today’s episode is somewhat special, because we’re going to be talking about what might be the first solid quantitative study of the power-seeking tendencies that we can expect advanced AI systems to have in the future. For a long time, there’s kind of been this debate in the AI safety world, between: People who worry that powerful AIs could eventually displace, or even eliminate humanity altogether as they find more clever, creative and dangerous ways to optimize their reward metrics on the one hand, and People who say that’s Terminator-bating Hollywood nonsense that anthropomorphizes machines in a way that’s unhelpful and misleading. Unfortunately, recent work in AI alignment — and in particular, a spotlighted 2021 NeurIPS paper — suggests that the AI takeover argument might be stronger than many had realized. In fact, it’s starting to look like we ought to expect to see power-seeking behaviours from highly capable AI systems by default. These behaviours include things like AI systems preventing us from shutting them down, repurposing resources in pathological ways to serve their objectives, and even in the limit, generating catastrophes that would put humanity at risk. As concerning as these possibilities might be, it’s exciting that we’re starting to develop a more robust and quantitative language to describe AI failures and power-seeking. That’s why I was so excited to sit down with AI researcher Alex Turner, the author of the spotlighted NeurIPS paper on power-seeking, and discuss his path into AI safety, his research agenda and his perspective on the future of AI on this episode of the TDS podcast. *** Intro music: ➞ Artist: Ron Gelinas ➞ Track Title: Daybreak Chill Blend (original mix) ➞ Link to Track: https://youtu.be/d8Y2sKIgFWc *** Chapters:  - 2:05 Interest in alignment research - 8:00 Two camps of alignment research - 13:10 The NeurIPS paper - 17:10 Optimal policies - 25:00 Two-piece argument - 28:30 Relaxing certain assumptions - 32:45 Objections to the paper - 39:00 Broader sense of optimization - 46:35 Wrap-up

Until recently, AI systems have been narrow — they’ve only been able to perform the specific tasks that they were explicitly trained for. And while narrow systems are clearly useful, the holy grain of AI is to build more flexible, general systems. But that can’t be done without good performance metrics that we can optimize for — or that we can at least use to measure generalization ability. Somehow, we need to figure out what number needs to go up in order to bring us closer to generally-capable agents. That’s the question we’ll be exploring on this episode of the podcast, with Danijar Hafner. Danijar is a PhD student in artificial intelligence at the University of Toronto with Jimmy Ba and Geoffrey Hinton and researcher at Google Brain and the Vector Institute. Danijar has been studying the problem of performance measurement and benchmarking for RL agents with generalization abilities. As part of that work, he recently released Crafter, a tool that can procedurally generate complex environments that are a lot like Minecraft, featuring resources that need to be collected, tools that can be developed, and enemies who need to be avoided or defeated. In order to succeed in a Crafter environment, agents need to robustly plan, explore and test different strategies, which allow them to unlock certain in-game achievements. Crafter is part of a growing set of strategies that researchers are exploring to figure out how we can benchmark and measure the performance of general-purpose AIs, and it also tells us something interesting about the state of AI: increasingly, our ability to define tasks that require the right kind of generalization abilities is becoming just as important as innovating on AI model architectures. Danijar joined me to talk about Crafter, reinforcement learning, and the big challenges facing AI researchers as they work towards general intelligence on this episode of the TDS podcast. *** Intro music: - Artist: Ron Gelinas - Track Title: Daybreak Chill Blend (original mix) - Link to Track: https://youtu.be/d8Y2sKIgFWc *** Chapters: 0:00 Intro 2:25 Measuring generalization 5:40 What is Crafter? 11:10 Differences between Crafter and Minecraft 20:10 Agent behavior 25:30 Merging scaled models and reinforcement learning 29:30 Data efficiency 38:00 Hierarchical learning 43:20 Human-level systems 48:40 Cultural overlap 49:50 Wrap-up

2021 has been a wild ride in many ways, but its wildest features might actually be AI-related. We’ve seen major advances in everything from language modeling to multi-modal learning, open-ended learning and even AI alignment. So, we thought, what better way to take stock of the big AI-related milestones we’ve reached in 2021 than a cross-over episode with our friends over at the Last Week In AI podcast. *** Intro music: - Artist: Ron Gelinas - Track Title: Daybreak Chill Blend (original mix) - Link to Track: https://youtu.be/d8Y2sKIgFWc *** Chapters: 0:00 Intro 2:15 Rise of multi-modal models 7:40 Growth of hardware and compute 13:20 Reinforcement learning 20:45 Open-ended learning 26:15 Power seeking paper 32:30 Safety and assumptions 35:20 Intrinsic vs. extrinsic motivation 42:00 Mapping natural language 46:20 Timnit Gebru’s research institute 49:20 Wrap-up

Imagine for a minute that you’re running a profitable business, and that part of your sales strategy is to send the occasional mass email to people who’ve signed up to be on your mailing list. For a while, this approach leads to a reliable flow of new sales, but then one day, that abruptly stops. What happened? You pour over logs, looking for an explanation, but it turns out that the problem wasn’t with your software; it was with your data. Maybe the new intern accidentally added a character to every email address in your dataset, or shuffled the names on your mailing list so that Christina got a message addressed to “John”, or vice-versa. Versions of this story happen surprisingly often, and when they happen, the cost can be significant: lost revenue, disappointed customers, or worse — an irreversible loss of trust. Today, entire products are being built on top of datasets that aren’t monitored properly for critical failures — and an increasing number of those products are operating in high-stakes situations. That’s why data observability is so important: the ability to  track the origin, transformations and characteristics of mission-critical data to detect problems before they lead to downstream harm. And it’s also why we’ll be talking to Kevin Hu, the co-founder and CEO of Metaplane, one of the world’s first data observability startups. Kevin has a deep understanding of data pipelines, and the problems that cap pop up if you they aren’t properly monitored. He joined me to talk about data observability, why it matters, and how it might be connected to responsible AI on this episode of the TDS podcast. Intro music: ➞ Artist: Ron Gelinas ➞ Track Title: Daybreak Chill Blend (original mix) ➞ Link to Track: https://youtu.be/d8Y2sKIgFWc 0:00 Chapters:  0:00 Intro 2:00 What is data observability? 8:20 Difference between a dataset’s internal and external characteristics 12:20 Why is data so difficult to log? 17:15 Tracing back models 22:00 Algorithmic analyzation of a date 26:30 Data ops in five years 33:20 Relation to cutting-edge AI work 39:25 Software engineering and startup funding 42:05 Problems on a smaller scale 46:40 Future data ops problems to solve 48:45 Wrap-up

Historically, AI systems have been slow learners. For example, a computer vision model often needs to see tens of thousands of hand-written digits before it can tell a 1 apart from a 3. Even game-playing AIs like DeepMind’s AlphaGo, or its more recent descendant MuZero, need far more experience than humans do to master a given game. So when someone develops an algorithm that can reach human-level performance at anything as fast as a human can, it’s a big deal. And that’s exactly why I asked Yang Gao to join me on this episode of the podcast. Yang is an AI researcher with affiliations at Berkeley and Tsinghua University, who recently co-authored a paper introducing EfficientZero: a reinforcement learning system that learned to play Atari games at the human-level after just two hours of in-game experience. It’s a tremendous breakthrough in sample-efficiency, and a major milestone in the development of more general and flexible AI systems. ---  Intro music: ➞ Artist: Ron Gelinas ➞ Track Title: Daybreak Chill Blend (original mix) ➞ Link to Track: https://youtu.be/d8Y2sKIgFWc --- Chapters:  - 0:00 Intro - 1:50 Yang’s background - 6:00 MuZero’s activity - 13:25 MuZero to EfficiantZero - 19:00 Sample efficiency comparison - 23:40 Leveraging algorithmic tweaks - 27:10 Importance of evolution to human brains and AI systems - 35:10 Human-level sample efficiency - 38:28 Existential risk from AI in China - 47:30 Evolution and language - 49:40 Wrap-up

There once was a time when AI researchers could expect to read every new paper published in the field on the arXiv, but today, that’s no longer the case. The recent explosion of research activity in AI has turned keeping up to date with new developments into a full-time job. Fortunately, people like YouTuber, ML PhD and sunglasses enthusiast Yannic Kilcher make it their business to distill ML news and papers into a digestible form for mortals like you and me to consume. I highly recommend his channel to any TDS podcast listeners who are interested in ML research — it’s a fantastic resource, and literally the way I finally managed to understand the Attention is All You Need paper back in the day. Yannic is joined me to talk about what he’s learned from years of following, reporting and doing AI research, including the trends, the challenges and the opportunities that he expects are going to shape the course of AI history in coming years. ---  Intro music: ➞ Artist: Ron Gelinas ➞ Track Title: Daybreak Chill Blend (original mix) ➞ Link to Track: https://youtu.be/d8Y2sKIgFWc ---  Chapters: - 0:00 Intro - 1:20 Yannic’s path into ML - 7:25 Selecting ML news - 11:45 AI ethics → political discourse - 17:30 AI alignment - 24:15 Malicious uses - 32:10 Impacts on persona - 39:50 Bringing in human thought - 46:45 Math with big numbers - 51:05 Metrics for generalization - 58:05 The future of AI - 1:02:58 Wrap-up

Today, most machine learning algorithms use the same paradigm: set an objective, and train an agent, a neural net, or a classical model to perform well against that objective. That approach has given good results: these types of AI can hear, speak, write, read, draw, drive and more. But they’re also inherently limited: because they optimize for objectives that seem interesting to humans, they often avoid regions of parameter space that are valuable, but that don’t immediately seem interesting to human beings, or the objective functions we set. That poses a challenge for researchers like Ken Stanley, whose goal is to build broadly superintelligent AIs — intelligent systems that outperform humans at a wide range of tasks. Among other things, Ken is a former startup founder and AI researcher, whose career has included work in academia, at UberAI labs, and most recently at OpenAI, where he leads the open-ended learning team. Ken joined me to talk about his 2015 book Greatness Cannot Be Planned: The Myth of the Objective, what open-endedness could mean for humanity, the future of intelligence, and even AI safety on this episode of the TDS podcast.

It’s no secret that governments around the world are struggling to come up with effective policies to address the risks and opportunities that AI presents. And there are many reasons why that’s happening: many people — including technical people — think they understand what frontier AI looks like, but very few actually do, and even fewer are interested in applying their understanding in a government context, where salaries are low and stock compensation doesn’t even exist. So there’s a critical policy-technical gap that needs bridging, and failing to address that gap isn’t really an option: it would mean flying blind through the most important test of technological governance the world has ever faced. Unfortunately, policymakers have had to move ahead with regulating and legislating with that dangerous knowledge gap in place, and the result has been less-than-stellar: widely criticized definitions of privacy and explainability, and definitions of AI that create exploitable loopholes are among some of the more concerning results. Enter Gillian Hadfield, a Professor of Law and Professor of Strategic Management and Director of the Schwartz Reisman Institute for Technology and Society. Gillian’s background is in law and economics, which has led her to AI policy, and definitional problems with recent and emerging regulations on AI and privacy. But — as I discovered during the podcast — she also happens to be related to Dyllan Hadfield-Menell, an AI alignment researcher whom we’ve had on the show before. Partly through Dyllan, Gillian has also been exploring how principles of AI alignment research can be applied to AI policy, and to contract law. Gillian joined me to talk about all that and more on this episode of the podcast. --- Intro music: - Artist: Ron Gelinas - Track Title: Daybreak Chill Blend (original mix) - Link to Track: https://youtu.be/d8Y2sKIgFWc  --- Chapters: 1:35 Gillian’s background 8:44 Layers and governments’ legislation 13:45 Explanations and justifications 17:30 Explainable humans 24:40 Goodhart’s Law 29:10 Bringing in AI alignment 38:00 GDPR 42:00 Involving technical folks 49:20 Wrap-up

AI ethics is often treated as a dry, abstract academic subject. It doesn’t have the kinds of consistent, unifying principles that you might expect from a quantitative discipline like computer science or physics. But somehow, the ethics rubber has to meet the AI road, and where that happens — where real developers have to deal with real users and apply concrete ethical principles — is where you find some of the most interesting, practical thinking on the topic. That’s why I wanted to speak with Wendy Foster, the Director of Engineering and Data Science at Shopify. Wendy’s approach to AI ethics is refreshingly concrete and actionable. And unlike more abstract approaches, it’s based on clear principles like user empowerment: the idea that you should avoid forcing users to make particular decisions, and instead design user interfaces that frame AI-recommended actions as suggestions that can be ignored or acted on. Wendy joined me to discuss her practical perspective on AI ethics, the importance of user experience design for AI products, and how responsible AI gets baked into product at Shopify on this episode of the TDS podcast. --- Intro music: - Artist: Ron Gelinas - Track Title: Daybreak Chill Blend (original mix) - Link to Track: https://youtu.be/d8Y2sKIgFWc --- Chapters:  - 0:00 Intro - 1:40 Wendy’s background - 4:40 What does practice mean? - 14:00 Different levels of explanation - 19:05 Trusting the system - 24:00 Training new folks - 30:02 Company culture - 34:10 The core of AI ethics - 40:10 Communicating with the user - 44:15 Wrap-up

Over the last two years, the capabilities of AI systems have exploded. AlphaFold2, MuZero, CLIP, DALLE, GPT-3 and many other models have extended the reach of AI to new problem classes. There’s a lot to be excited about. But as we’ve seen in other episodes of the podcast, there’s a lot more to getting value from an AI system than jacking up its capabilities. And increasingly, one of these additional missing factors is becoming trust. You can make all the powerful AIs you want, but if no one trusts their output — or if people trust it when they shouldn’t — you can end up doing more harm than good. That’s why we invited Ayanna Howard on the podcast. Ayanna is a roboticist, entrepreneur and Dean of the College of Engineering at Ohio State University, where she focuses her research on human-machine interactions and the factors that go into building human trust in AI systems. She joined me to talk about her research, its applications in medicine and education, and the future of human-machine trust. --- Intro music: - Artist: Ron Gelinas - Track Title: Daybreak Chill Blend (original mix) - Link to Track: https://youtu.be/d8Y2sKIgFWc --- Chapters: - 0:00 Intro - 1:30 Ayanna’s background - 6:10 The interpretability of neural networks - 12:40 Domain of machine-human interaction - 17:00 The issue of preference - 20:50 Gelman/newspaper amnesia - 26:35 Assessing a person’s persuadability - 31:40 Doctors and new technology - 36:00 Responsibility and accountability - 43:15 The social pressure aspect - 47:15 Is Ayanna optimistic? - 53:00 Wrap-up