Epoch AI is the premier organization that tracks the trajectory of AI - how much compute is used, the role of algorithmic improvements, the growth in data used, and when the above trends might hit an end. In this episode, I speak with the director of Epoch AI, Jaime Sevilla, about how compute, data, and algorithmic improvements are impacting AI, and whether continuing to scale can get us AGI.
Patreon: https://www.patreon.com/axrpodcast
Ko-fi: https://ko-fi.com/axrpodcast
The transcript: https://axrp.net/episode/2024/10/04/episode-37-jaime-sevilla-forecasting-ai.html
Topics we discuss, and timestamps:
0:00:38 - The pace of AI progress
0:07:49 - How Epoch AI tracks AI compute
0:11:44 - Why does AI compute grow so smoothly?
0:21:46 - When will we run out of computers?
0:38:56 - Algorithmic improvement
0:44:21 - Algorithmic improvement and scaling laws
0:56:56 - Training data
1:04:56 - Can scaling produce AGI?
1:16:55 - When will AGI arrive?
1:21:20 - Epoch AI
1:27:06 - Open questions in AI forecasting
1:35:21 - Epoch AI and x-risk
1:41:34 - Following Epoch AI's research
Links for Jaime and Epoch AI:
Epoch AI: https://epochai.org/
Machine Learning Trends dashboard: https://epochai.org/trends
Epoch AI on X / Twitter: https://x.com/EpochAIResearch
Jaime on X / Twitter: https://x.com/Jsevillamol
Research we discuss:
Training Compute of Frontier AI Models Grows by 4-5x per Year: https://epochai.org/blog/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year
Optimally Allocating Compute Between Inference and Training: https://epochai.org/blog/optimally-allocating-compute-between-inference-and-training
Algorithmic Progress in Language Models [blog post]: https://epochai.org/blog/algorithmic-progress-in-language-models
Algorithmic progress in language models [paper]: https://arxiv.org/abs/2403.05812
Training Compute-Optimal Large Language Models [aka the Chinchilla scaling law paper]: https://arxiv.org/abs/2203.15556
Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data [blog post]: https://epochai.org/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data
Will we run out of data? Limits of LLM scaling based on human-generated data [paper]: https://arxiv.org/abs/2211.04325
The Direct Approach: https://epochai.org/blog/the-direct-approach
Episode art by Hamish Doodles: hamishdoodles.com
