Signals and Threads The Uncertain Art of Accelerating ML Models with Sylvain Gugger
35 snips
Oct 14, 2024 Sylvain Gugger, a machine learning engineer at Jane Street and co-author of "Deep Learning for Coders," shares his fascinating transition from teaching math to ML. He delves into optimizing learning rate strategies and the nuances of working with PyTorch. The conversation touches on the importance of reproducibility in training models as well as the challenges of inference in trading, emphasizing low latency with uniquely shaped market data. Sylvain also highlights Hugging Face's role in making ML tools more accessible, enhancing collaboration within the field.
AI Snips
Chapters
Transcript
Episode notes
DAWNBench Competition
- Sylvain Gugger and Fast.ai entered the DAWNBench competition at Stanford, aiming to train a vision model quickly.
- Google ultimately won using newly released TPUs, outperforming their leading entry.
Learning Rate Schedules
- Learning rate schedules significantly impact model training speed.
- A warm-up period, increasing and then decreasing the learning rate, improves training efficiency.
Gradual Resizing
- Sylvain Gugger's team used gradual resizing of images during training.
- Starting with smaller images and increasing size improved the efficiency of their convolutional neural network (CNN).