Guanhua Wang, a Senior Researcher in the DeepSpeed team at Microsoft, dives into the revolutionary Domino training engine, designed to eliminate communication overhead during LLM training. He discusses the intricacies of naming the Phi-3 model and the growing interest in smaller language models. Wang highlights advanced techniques like data offloading and quantization, showcasing how Domino can speed up training by up to 1.3x compared to existing methods, while addressing privacy in customizable copilot models. It's a deep dive into optimizing AI training!
49:47
forum Ask episode
web_stories AI Snips
view_agenda Chapters
auto_awesome Transcript
info_circle Episode notes
question_answer ANECDOTE
Phi-3's Ambitious Goal
Guanhua Wang, from Microsoft's DeepSpeed team, discussed Phi-3, a small language model designed to mimic physical environments.
The project aimed to enable LLMs to reflect and act within these environments, but development stalled after Phi-3.
insights INSIGHT
Data Quality Matters
High-quality, minimally noisy data is crucial for training small language models effectively.
Investing in top-tier data sources, like The New York Times or Forbes, significantly improves model performance.
insights INSIGHT
DeepSpeed's Efficiency
DeepSpeed, a PyTorch-based library, offers features like Zero optimizer for memory-efficient data parallel training.
It shards model parameters across GPUs and offloads data to the CPU to reduce memory pressure.
Get the Snipd Podcast app to discover more snips from this episode
Guanhua Wang is a Senior Researcher in DeepSpeed Team at Microsoft. Before Microsoft, Guanhua earned his Computer Science PhD from UC Berkeley.
Domino: Communication-Free LLM Training Engine // MLOps Podcast #278 with Guanhua "Alex" Wang, Senior Researcher at Microsoft.
// Abstract
Given the popularity of generative AI, Large Language Models (LLMs) often consume hundreds or thousands of GPUs to parallelize and accelerate the training process. Communication overhead becomes more pronounced when training LLMs at scale. To eliminate communication overhead in distributed LLM training, we propose Domino, which provides a generic scheme to hide communication behind computation. By breaking the data dependency of a single batch training into smaller independent pieces, Domino pipelines these independent pieces of training and provides a generic strategy of fine-grained communication and computation overlapping. Extensive results show that compared with Megatron-LM, Domino achieves up to 1.3x speedup for LLM training on Nvidia DGX-H100 GPUs.
// Bio
Guanhua Wang is a Senior Researcher in the DeepSpeed team at Microsoft. His research focuses on large-scale LLM training and serving. Previously, he led the ZeRO++ project at Microsoft which helped reduce over half of model training time inside Microsoft and Linkedin. He also led and was a major contributor to Microsoft Phi-3 model training. He holds a CS PhD from UC Berkeley advised by Prof Ion Stoica.
// MLOps Swag/Merch
https://shop.mlops.community/
// Related Links
Website: https://guanhuawang.github.io/
DeepSpeed hiring: https://www.microsoft.com/en-us/research/project/deepspeed/opportunities/
Large Model Training and Inference with DeepSpeed // Samyam Rajbhandari // LLMs in Prod Conference: https://youtu.be/cntxC3g22oU
--------------- ✌️Connect With Us ✌️ -------------
Join our slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register
Catch all episodes, blogs, newsletters, and more: https://mlops.community/
Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/
Connect with Guanhua on LinkedIn: https://www.linkedin.com/in/guanhua-wang/
Timestamps:
[00:00] Guanhua's preferred coffee
[00:17] Takeaways
[01:36] Please like, share, leave a review, and subscribe to our MLOps channels!
[01:47] Phi model explanation
[06:29] Small Language Models optimization challenges
[07:29] DeepSpeed overview and benefits
[10:58] Crazy unimplemented crazy AI ideas
[17:15] Post training vs QAT
[19:44] Quantization over distillation
[24:15] Using Lauras
[27:04] LLM scaling sweet spot
[28:28] Quantization techniques
[32:38] Domino overview
[38:02] Training performance benchmark
[42:44] Data dependency-breaking strategies
[49:14] Wrap up
MLOps.community