How to Predict Power Laws on Tokens

Eric is a PhD student in the Department of Physics at MIT working with Max Tegmark on improving our scientific/theoretical understanding of deep learning -- understanding what deep neural networks do internally and why they work so well. This is part of a broader interest in the nature of intelligent systems, which previously led him to work with SETI astronomers, with Stuart Russell's AI alignment group (CHAI), and with Erik Hoel on a project related to integrated information theory.

Transcript: https://theinsideview.ai/eric

Youtube: https://youtu.be/BtHMIQs_5Nw

The Quantization Model of Neural Scaling: https://arxiv.org/abs/2303.13506
An Effective Theory of Representation Learning https://arxiv.org/abs/2205.10343

Omnigrok: Grokking Beyond Algorithmic Data: https://arxiv.org/abs/2210.01117