LessWrong (Curated & Popular) “What o3 Becomes by 2028” by Vladimir_Nesov
4 snips
Jan 9, 2025 Vladimir Nesov, author known for his insights on AI, dives deep into the future of OpenAI's O3 training systems. He discusses the significance of upcoming investments in scaling AI capabilities with models projected to train at unprecedented FLOP levels. The conversation highlights the balance between data quality and quantity, emphasizing the need for 50 trillion training tokens. Nesov also evaluates the current state of GPT-4 and its competitors, pondering what advancements might emerge by 2028 in this rapidly evolving landscape.
AI Snips
Chapters
Transcript
Episode notes
GPT-4's Reign
- GPT-4's capabilities remained unmatched throughout 2024, likely due to its substantial computational resources and model architecture.
- Competitors caught up only by replicating similar scale and expertise, highlighting the importance of both factors in AI development.
Scaling Engines
- In 2024, large training systems emerged, costing billions and requiring significant power, demonstrating increased investment in AI.
- Hardware advantages like larger scale-up domains and smaller minibatch sizes might give some labs an edge in scaling.
Future Scaling
- Funding for massive training systems is currently feasible but might become challenging without stronger justification.
- OpenAI's O3 and its successors could influence the decision to build even larger, more expensive systems.