Lex Fridman Podcast cover image

#459 – DeepSeek, China, OpenAI, NVIDIA, xAI, TSMC, Stargate, and AI Megaclusters

Lex Fridman Podcast

00:00

Transformer Memory Cost

  • The attention operator in transformers has a quadratic memory cost relative to context length.
  • Longer inputs significantly increase memory usage, leading to innovations like subquadratic and linear attention forms.
Transcript
Play full episode

The AI-powered Podcast Player

Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
App store bannerPlay store banner
Get the app