The Limits of Scaling Up AI Models

OpenAI says its estimates suggest diminishing returns on scaling up model size. Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them. Nick Frost, a co-founder at cohere who previously worked on AI at Google, says Altman's feeling that going bigger will not work is true. He too believes the progress on transformers lies beyond scaling.