Training Data Meta’s Joe Spisak on Llama 3.1 405B and the Democratization of Frontier Models
17 snips
Jul 30, 2024 Joe Spisak is the head of Product Management for Generative AI at Meta, leading the team behind the recently launched Llama 3.1 405B model. He discusses how this model, trained on 15 trillion tokens, enhances zero-shot tool use and aids in generating synthetic data. Spisak believes frontier models will commoditize, benefiting startups and promoting open-source innovation. The conversation also touches on the integration of coding to improve AI reasoning, the advantages of smaller models, and the balance between open and closed systems in AI development.
AI Snips
Chapters
Transcript
Episode notes
Llama 3.1: A Teacher for Models
- The Llama 3.1 405B model serves as a "massive teacher" for smaller models through distillation and synthetic data.
- Joe Spisak describes it as a "monster" model with great multilingual and safety capabilities developed after long internal work.
Zero-Shot Tool Use Breakthrough
- Zero-shot tool use is a major new capability enabled by Llama 3.1 405B with calls to Wolfram, Google Search, and code interpreters.
- This ability will be a game changer for the community and product developers.
Frontier Models Commoditize Quickly
- Models at the frontier are rapidly commoditizing; the value lies in the product built on top of them.
- Scale on data and compute yields first order approximations to state of the art, but ultimate value is in application integration.