Tri Dao, an incoming professor at Princeton and Chief Scientist at Together AI, joins Michael Poli, a Stanford PhD graduate and research scientist at Together AI. They dive into why traditional attention mechanisms may not scale effectively and introduce innovative models like Striped Hyena and Mamba. The duo discusses hardware optimization for these architectures and predicts exciting developments in AI for 2024, challenging the dominance of current transformer models. Their insights reflect a transformative wave in machine learning.