“Contra papers claiming superhuman AI forecasting ” by nikos, Peter Mühlbacher, Lawrence Phillips, dschwarz

[Conflict of interest disclaimer: We are FutureSearch, a company working on AI-powered forecasting and other types of quantitative reasoning. If thin LLM wrappers could achieve superhuman forecasting performance, this would obsolete a lot of our work.]

Widespread, misleading claims about AI forecasting

Recently we have seen a number of papers – (Schoenegger et al., 2024, Halawi et al., 2024, Phan et al., 2024, Hsieh et al., 2024) – with claims that boil down to “we built an LLM-powered forecaster that rivals human forecasters or even shows superhuman performance”.

These papers do not communicate their results carefully enough, shaping public perception in inaccurate and misleading ways. Some examples of public discourse:

• Ethan Mollick (>200k followers) tweeted the following about the paper Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy by Schoenegger et al.:
  
  
• A post on Marginal Revolution with the title and abstract [...]

---

Outline:

(00:24) Widespread, misleading claims about AI forecasting

(03:02) What does human-level or superhuman forecasting mean?

(04:08) Red flags for claims to (super)human AI forecasting accuracy

(06:42) Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy (Schoenegger et al., 2024)

(09:14) Approaching Human-Level Forecasting with Language Models (Halawi et al., 2024)

(11:10) Reasoning and Tools for Human-Level Forecasting (Hsieh et al., 2024)

(12:50) LLMs Are Superhuman Forecasters (Phan et al., 2024)

(15:19) Takeaways

(16:17) So how good are AI forecasters?

The original text contained 1 footnote which was omitted from this narration.

The original text contained 6 images which were described by AI.

---

First published:
September 12th, 2024

Source:
https://www.lesswrong.com/posts/uGkRcHqatmPkvpGLq/contra-papers-claiming-superhuman-ai-forecasting

---

Narrated by TYPE III AUDIO.

---