How seriously should governments take the threat of existential risk from AI, given the lack of consensus among researchers? On the one hand, existential risks (x-risks) are necessarily somewhat speculative: by the time there is concrete evidence, it may be too late. On the other hand, governments must prioritize — after all, they don’t worry too much about x-risk from alien invasions.
MLST is sponsored by Brave:
The Brave Search API covers over 20 billion webpages, built from scratch without Big Tech biases or the recent extortionate price hikes on search API access. Perfect for AI model training and retrieval augmentated generation. Try it now - get 2,000 free queries monthly at brave.com/api.
Sayash Kapoor is a computer science Ph.D. candidate at Princeton University's Center for Information Technology Policy. His research focuses on the societal impact of AI. Kapoor has previously worked on AI in both industry and academia, with experience at Facebook, Columbia University, and EPFL Switzerland. He is a recipient of a best paper award at ACM FAccT and an impact recognition award at ACM CSCW. Notably, Kapoor was included in TIME's inaugural list of the 100 most influential people in AI.
Sayash Kapoor
https://x.com/sayashk
https://www.cs.princeton.edu/~sayashk/
Arvind Narayanan (other half of the AI Snake Oil duo)
https://x.com/random_walker
AI existential risk probabilities are too unreliable to inform policy
https://www.aisnakeoil.com/p/ai-existential-risk-probabilities
Pre-order AI Snake Oil Book
https://amzn.to/4fq2HGb
AI Snake Oil blog
https://www.aisnakeoil.com/
AI Agents That Matter
https://arxiv.org/abs/2407.01502
Shortcut learning in deep neural networks
https://www.semanticscholar.org/paper/Shortcut-learning-in-deep-neural-networks-Geirhos-Jacobsen/1b04936c2599e59b120f743fbb30df2eed3fd782
77% Of Employees Report AI Has Increased Workloads And Hampered Productivity, Study Finds
https://www.forbes.com/sites/bryanrobinson/2024/07/23/employees-report-ai-increased-workload/
TOC:
00:00:00 Intro
00:01:57 How seriously should we take Xrisk threat?
00:02:55 Risk too unrealiable to inform policy
00:10:20 Overinflated risks
00:12:05 Perils of utility maximisation
00:13:55 Scaling vs airplane speeds
00:17:31 Shift to smaller models?
00:19:08 Commercial LLM ecosystem
00:22:10 Synthetic data
00:24:09 Is AI complexifying our jobs?
00:25:50 Does ChatGPT make us dumber or smarter?
00:26:55 Are AI Agents overhyped?
00:28:12 Simple vs complex baselines
00:30:00 Cost tradeoff in agent design
00:32:30 Model eval vs downastream perf
00:36:49 Shortcuts in metrics
00:40:09 Standardisation of agent evals
00:41:21 Humans in the loop
00:43:54 Levels of agent generality
00:47:25 ARC challenge
Machine Learning Street Talk (MLST) 
