In this book, Tetlock and Gardner present the results of the Good Judgment Project, a massive forecasting tournament that identified a small group of 'superforecasters' who are exceptionally good at predicting future events. The authors explain that good forecasting does not require powerful computers or arcane methods but involves gathering evidence from various sources, thinking probabilistically, working in teams, keeping score, and being willing to admit error and change course. The book uses stories of forecasting successes and failures, as well as interviews with high-level decision makers, to illustrate these principles and demonstrate how anyone can improve their forecasting abilities[3][4][5].
Douglas Hubbard's "How to Measure Anything" provides a practical framework for quantifying even the seemingly unmeasurable. The book emphasizes the importance of understanding uncertainty and using Bayesian methods to improve decision-making. Hubbard presents a systematic approach to identifying and addressing sources of uncertainty, helping readers make more informed choices. The book is widely used by professionals in various fields, including business, engineering, and science. It offers valuable insights into the challenges of measurement and the importance of using data effectively to improve decision-making.
Companies use about 300,000 times more computation training the best AI systems today than they did in 2012 and algorithmic innovations have also made them 25 times more efficient at the same tasks.
These are the headline results of two recent papers — AI and Compute and AI and Efficiency — from the Foresight Team at OpenAI. In today's episode I spoke with one of the authors, Danny Hernandez, who joined OpenAI after helping develop better forecasting methods at Twitch and Open Philanthropy.
Danny and I talk about how to understand his team's results and what they mean (and don't mean) for how we should think about progress in AI going forward.
Links to learn more, summary and full transcript.
Debates around the future of AI can sometimes be pretty abstract and theoretical. Danny hopes that providing rigorous measurements of some of the inputs to AI progress so far can help us better understand what causes that progress, as well as ground debates about the future of AI in a better shared understanding of the field.
If this research sounds appealing, you might be interested in applying to join OpenAI's Foresight team — they're currently hiring research engineers.
In the interview, Danny and I (Arden Koehler) also discuss a range of other topics, including:
• The question of which experts to believe
• Danny's journey to working at OpenAI
• The usefulness of "decision boundaries"
• The importance of Moore's law for people who care about the long-term future
• What OpenAI's Foresight Team's findings might imply for policy
• The question whether progress in the performance of AI systems is linear
• The safety teams at OpenAI and who they're looking to hire
• One idea for finding someone to guide your learning
• The importance of hardware expertise for making a positive impact
Chapters:
- Rob’s intro (00:00:00)
- The interview begins (00:01:29)
- Forecasting (00:07:11)
- Improving the public conversation around AI (00:14:41)
- Danny’s path to OpenAI (00:24:08)
- Calibration training (00:27:18)
- AI and Compute (00:45:22)
- AI and Efficiency (01:09:22)
- Safety teams at OpenAI (01:39:03)
- Careers (01:49:46)
- AI hardware as a possible path to impact (01:55:57)
- Triggers for people’s major decisions (02:08:44)
Producer: Keiran Harris
Audio mastering: Ben Cordell
Transcriptions: Zakee Ulhaq
80,000 Hours Podcast 
