Episode 40: DeepSeek facts vs hype, model distillation, and open source competition
Mixture of Experts
00:00
Unpacking Pre-Training Costs in Deep Learning
This chapter explores the misconceptions about the expenses related to training large deep learning models, revealing that true costs are often underestimated due to hidden factors. It emphasizes the role of open-source models and reinforcement learning in reducing barriers for startups, while also highlighting the extensive experimentation required to achieve successful outcomes.
Transcript
Play full episode
