Coercing LLMs to Do and Reveal (Almost) Anything with Jonas Geiping - #678
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
Optimizing with Hybrid Strategies
The speaker discusses the limitations of straightforward gradient searches for complex models like llama two models. They introduce the concept of introducing randomness in gradient-based optimizers which enhances the optimization process. The G2G algorithm involves collecting a list of 256 largest magnitude candidates and selecting randomly from them, creating a hybrid approach between random search and gradient-based optimization. This method helps overcome barriers in the optimization landscape by introducing structured randomness, which differs from traditional gradient-based approaches.
00:00
Transcript
Play full episode
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
