MLOps.community LLM Distillation and Compression // Guanhua Wang // #278
8 snips
Dec 17, 2024 Guanhua Wang, a Senior Researcher in the DeepSpeed team at Microsoft, dives into the revolutionary Domino training engine, designed to eliminate communication overhead during LLM training. He discusses the intricacies of naming the Phi-3 model and the growing interest in smaller language models. Wang highlights advanced techniques like data offloading and quantization, showcasing how Domino can speed up training by up to 1.3x compared to existing methods, while addressing privacy in customizable copilot models. It's a deep dive into optimizing AI training!
AI Snips
Chapters
Transcript
Episode notes
Phi-3's Ambitious Goal
- Guanhua Wang, from Microsoft's DeepSpeed team, discussed Phi-3, a small language model designed to mimic physical environments.
- The project aimed to enable LLMs to reflect and act within these environments, but development stalled after Phi-3.
Data Quality Matters
- High-quality, minimally noisy data is crucial for training small language models effectively.
- Investing in top-tier data sources, like The New York Times or Forbes, significantly improves model performance.
DeepSpeed's Efficiency
- DeepSpeed, a PyTorch-based library, offers features like Zero optimizer for memory-efficient data parallel training.
- It shards model parameters across GPUs and offloads data to the CPU to reduce memory pressure.