AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
5min chapter
Hattie Zhou: Lottery Tickets and Algorithmic Reasoning in LLMs
The Gradient: Perspectives on AI
In-Context Learning and Gradient Decading
There is a cool natural link there. Another series of papers that I found kind of neat recently has been the ones that are trying to establish this relationship between in-context learning and gradient descent. In context learning, it feels like, well, literally, you're not updating the weights of your pre-trained model anymore. But some of these works did this really interesting set of derivations to look at like a dual form between gradient descent and then linear self-attention. And they understand in context learning as it's kind of implicit fine-tuning where the transformer attention is like doing a meta optimization. Then in its forward pass, it basically is implementing gradient descent.
00:00
Transcript
Episode notes
Get the Snipd
podcast app
Unlock the knowledge in podcasts with the podcast player of the future.
AI-powered
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode
Save any
moment
Hear something you like? Tap your headphones to save it with AI-generated key takeaways
Share
& Export
Send highlights to Twitter, WhatsApp or export them to Notion, Readwise & more
AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Discover
highlights
Listen to the best highlights from the podcasts you love and dive into the full episode