"The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis cover image

Imbue CTO Josh Albrecht on Creating AI Agents for Reasoning, Reliability, and Robustness

"The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis

00:00

How big is GPT-4 and how long does it take to train?

Estimating the size of GPT-4, I assume it to be 10 to the power of 25 in terms of flops. This is based on the recent declaration of a reporting threshold at 10 to the power of 26. Additionally, I consider the device flops to be four times 10 to the power of 15, assuming eight-digit quantization in training. To estimate the time for GPT-4, I refer to a source suggesting it took approximately 30,000 A100s for three to five months. While the specifics are unclear, this estimate aligns with the number of parameters and experts in the model.

Transcript
Play full episode

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
App store bannerPlay store banner
Get the app