Using Distributed Computing to Train Large Models

Have suggestions for future podcast guests (or other feedback)? Let us know here!

In episode 42 of The Gradient Podcast, Daniel Bashir speaks to Andrew Feldman.

Andrew is the co-founder and CEO of Cerebras Systems, an AI accelerator company that has built the largest processor in the industry. Before Cerebras, Andrew co-founded and served as CEO of SeaMicro, which was acquired by AMD in 2012. He has also served in executive positions at Force10 Networks and RiverStone Networks.

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on Twitter

Outline:

* (00:00) Intro

* (02:05) Andrew’s trajectory, from business school to Cerebras

* (10:00) The large model problem and Cerebras’ approach

* (19:50) Cerebras’s GPT-J announcement

* (22:20) Andrew explains weight streaming to Daniel

* (32:30) Andrew’s thoughts on the MLPerf benchmark

* (38:20) The venture landscape for AI accelerator companies

* (42:50) The hardware lottery, hardware support for sparsity

* (45:40) The CHIPS Act, NVIDIA China ban and the accelerator industry

* (48:00) Politics and Chips, US and China

* (52:20) Andrew’s perspective on tackling difficult problems

* (56:42) Outro

Links:

* Cerebras’ Homepage

* GPT-J Announcement

* TotalEnergies

* GlaxoSmithKline (GSK)

* Sources mentioned

* “Political Chips” by Ben Thompson (because Daniel’s a fanboy)

* Daniel’s conversation with Sara Hooker

* The Hardware Lottery