Interconnects

Phi 3 and Arctic: Outlier LMs are hints

Apr 30, 2024
The podcast discusses the future of AI industry with outlier LLMs, focusing on Phi 3 and Arctic models. It explores trends in open mixture of expert models and the impact of synthetic data on small models. The episode also touches on the training techniques of the new models, including Arctic's sparse mixture of experts architecture and Phi 3's evolution in the LLM space.
Ask episode
Chapters
Transcript
Episode notes
1
Introduction
00:00 • 4min
2
Exploring Residual Streams, Expert Models, and Potential Outliers in LM Space
04:23 • 2min
3
Evolution of Small Language Models and Critiques of Phi Team
06:23 • 3min