The Implications of Scaling for AGI

Scaling LLMs++ can potentially lead to powerful AI's by 2040 or sooner, automating cognitive labor and accelerating AI progress. However, the lack of sufficient high-quality language data could present a challenge to the scaling process. The need for exponentially increasing data and the difficulty of implementing self-play synthetic data pose further obstacles. Yet, proponents believe that the generous improvements expected from techniques like multi-modal training and curriculum learning might contribute, although not significantly enough, to match the exponential increase in compute demanded by scaling. Meanwhile, skeptics argue that the data bottleneck, coupled with the failure to increase models' underlying abilities with reinforcement learning, may significantly impede progress.