"The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis

Opening AI's Black Box with Prof. David Bau, Koyena Pal, and Eric Todd of Northeastern University

14 snips
Apr 5, 2024
Guest
Koyena Pal
Koyena Pal, a graduate student at Northeastern University and lead author of the Future Lens paper, joins Professor David Bau and Eric Todd to unpack the mysteries of large language models. Koyena reveals how mid-sized models think multiple tokens ahead, while Eric introduces the intriguing concept of Function Vectors that enable in-context learning. The trio discusses advancements in interpretability, the significance of hidden states, and the complexities of newer AI architectures, providing a fascinating insight into how these models process and predict information.
Ask episode
AI Snips
Chapters
Transcript
Episode notes
INSIGHT

Reverse Engineering AI

  • Machine learning models are like biological organisms, emerging from training.
  • Researchers must reverse-engineer their internals, similar to biologists with tomatoes.
INSIGHT

A Shift in Focus

  • David Bau's research focus shifted around 2015, anticipating changes in machine learning.
  • The success of AlexNet in 2012, using a simple neural network, was a key turning point.
ANECDOTE

ROME Paper Anecdote

  • The ROME paper explored localizing factual knowledge within language models.
  • It demonstrated editing facts while preserving model properties, like changing Michael Jordan's sport.
Get the Snipd Podcast app to discover more snips from this episode
Get the app