Tri Dao

Tri Dao, a recent Stanford PhD grad and Chief Scientist at Together AI, discusses his groundbreaking work on FlashAttention-2, enhancing transformer models for faster inference. He explains how FlashAttention improves efficiency by reducing memory usage from quadratic to linear scaling. The conversation also touches on the importance of memory architecture in GPU performance and the balance of traditional techniques with modern AI innovations. Lastly, Tri reflects on the dynamic landscape of AI research and the rise of open-source contributions in the field.

Tri Dao is a PhD student at Stanford, co-advised by Stefano Ermon and Chris Re. He’ll be joining Princeton as an assistant professor next year. He works at the intersection of machine learning and systems, currently focused on efficient training and long-range context. About Generally Intelligent  We started Generally Intelligent because we believe that software with human-level intelligence will have a transformative impact on the world. We’re dedicated to ensuring that that impact is a positive one.   We have enough funding to freely pursue our research goals over the next decade, and our backers include Y Combinator, researchers from OpenAI, Astera Institute, and a number of private individuals who care about effective altruism and scientific research.   Our research is focused on agents for digital environments (ex: browser, desktop, documents), using RL, large language models, and self supervised learning. We’re excited about opportunities to use simulated data, network architecture search, and good theoretical understanding of deep learning to make progress on these problems. We take a focused, engineering-driven approach to research.   Learn more about us Website: https://generallyintelligent.com/ LinkedIn: linkedin.com/company/generallyintelligent/  Twitter: @genintelligent

Tri Dao, an incoming professor at Princeton and Chief Scientist at Together AI, joins Michael Poli, a Stanford PhD graduate and research scientist at Together AI. They dive into why traditional attention mechanisms may not scale effectively and introduce innovative models like Striped Hyena and Mamba. The duo discusses hardware optimization for these architectures and predicts exciting developments in AI for 2024, challenging the dominance of current transformer models. Their insights reflect a transformative wave in machine learning.