Jeff Dean & Noam Shazeer – 25 years at Google: from PageRank to AGI

781 snips

Feb 12, 2025

Guest

Noam Shazeer

Guest

Jeff Dean

In this enlightening discussion, Jeff Dean, Google's Chief Scientist with 25 years of transformative tech experience, teams up with Noam Shazeer, a visionary behind key architectures in AI. They delve into the evolution of AI from basic n-grams to advanced neural networks, and the importance of AI-driven breakthroughs in chip design. Jeff shares his ambitious vision for Pathways, a revolutionary concept for optimizing algorithms and hardware. Noam boldly predicts a future with a multitude of automated researchers, hinting at a world forever transformed by technology.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

Early Mentorship

Jeff Dean mentored Noam Shazeer when he joined Google.
Shazeer was impressed by Dean's knowledge of Google's systems.

ANECDOTE

Joining Google

Jeff Dean actively sought out Google, while Noam Shazeer stumbled upon it at a job fair.
Shazeer initially dismissed Google, assuming it was too big, but later joined.

INSIGHT

Hardware's Influence

Deep learning's rise is linked to cheap arithmetic and expensive data movement.
This cost dynamic shaped algorithm design towards matrix multiplications.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

This week I welcome on the show two of the most important technologists ever, in any field.

Jeff Dean is Google's Chief Scientist, and through 25 years at the company, has worked on basically the most transformative systems in modern computing: from MapReduce, BigTable, Tensorflow, AlphaChip, to Gemini.

Noam Shazeer invented or co-invented all the main architectures and techniques that are used for modern LLMs: from the Transformer itself, to Mixture of Experts, to Mesh Tensorflow, to Gemini and many other things.

We talk about their 25 years at Google, going from PageRank to MapReduce to the Transformer to MoEs to AlphaChip – and maybe soon to ASI.

My favorite part was Jeff's vision for Pathways, Google’s grand plan for a mutually-reinforcing loop of hardware and algorithmic design and for going past autoregression. That culminates in us imagining *all* of Google-the-company, going through one huge MoE model.

And Noam just bites every bullet: 100x world GDP soon; let’s get a million automated researchers running in the Google datacenter; living to see the year 3000.Watch on Youtube; listen on Apple Podcasts or Spotify.