Consulting, startups, and AI Incident Database

Sean McGregor and I discuss about why evaluating AI systems has become so difficult; we cover everything from the breakdown of benchmarking, how incentives shape safety work, and what approaches like BenchRisk (his recent paper at NeurIPS) and AI auditing aim to fix as systems move into the real world. We also talk about his history and journey in AI safety, including his PhD on ML for public policy, how he started the AI Incident Database, and what he's working on now: AVERI, a non-profit for frontier model auditing.

Chapters

• (00:00) - Intro
• (02:36) - What's broken about benchmarking
• (03:41) - Sean’s wild PhD
• (14:28) - The phantom internship
• (19:25) - Sean's journey
• (22:25) - Market-vs-regulatory modes and AIID
• (32:13) - Drunk on AI progress
• (38:34) - BenchRisk
• (43:20) - Moral hazards and Master Hand
• (50:34) - Liability, Section 230, and open source
• (59:20) - AVERI
• (01:11:30) - Closing thoughts & outro

• Sean McGregor's website
• AVERI website

BenchRisk

• BenchRisk website
• NeurIPS paper - Risk Management for Mitigating Benchmark Failure Modes: BenchRisk
• NeurIPS paper - AI and the Everything in the Whole Wide World Benchmark

AIID

• AI Incident Database website
• IAAI paper - Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database
• Preprint - Lessons for Editors of AI Incidents from the AI Incident Database
• AIAAIC website (another incident tracker)

Hot AI Summer

• CACM article - A Few Useful Things to Know About Machine Learning
• CACM article - How the AI Boom Went Bust
• Undergraduate Thesis - Analyzing the Prospect of an Approaching AI Winter
• Tech Genies article - AI History: The First Summer and Winter of AI
• CACM article - There Was No ‘First AI Winter’

Measuring Generalization

• Neural Computation article - The Lack of A Priori Distinctions Between Learning Algorithms
• ICLR paper - Understanding deep learning requires rethinking generalization
• ICML paper - Model-agnostic Measure of Generalization Difficulty
• Radiology Artificial Intelligence article - Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls
• Preprint - Quantifying Generalization Complexity for Large Language Models

Insurers Exclude AI

• Financial Times article - Insurers retreat from AI cover as risk of multibillion-dollar claims mount
• Tom's Hardware article - Major insurers move to avoid liability for AI lawsuits as multi-billion dollar risks emerge — Recent public incidents have lead to costly repercussions
• Insurance Newsnet article - Insurers Scale Back AI Coverage Amid Fears of Billion-Dollar Claims
• Insurance Business article - Insurance’s gen AI reckoning has come

Section 230

• Section 230 overview
• Legal sidebar - Section 230 Immunity and Generative Artificial Intelligence
• Bad Internet Bills website
• TechDirt article - Section 230 Faces Repeal. Support The Coverage That’s Been Getting It Right All Along.
• Privacy Guides video - Dissecting Bad Internet Bills with Taylor Lorenz: KOSA, SCREEN Act, Section 230
• Journal of Technology in Behavioral Health article - Social Media and Mental Health: Benefits, Risks, and Opportunities for Research and Practice
• Time article - Lawmakers Unveil New Bills to Curb Big Tech’s Power and Profit
• House Hearing transcript - Legislative Solutions to Protect Children and Teens Online

Relevant Kairos.fm Episodes

• Into AI Safety episode - Growing BlueDot's Impact w/ Li-Lian Ang
• muckrAIkers episode - NeurIPS 2024 Wrapped 🌯

Other Links

• Encyclopedia of Life website
• IBM Watson AI XPRIZE website
• ML Commons website
• Wikipedia article