Machine Learning Street Talk (MLST)

Sepp Hochreiter - LSTM: The Comeback Story?

Feb 12, 2025

Sepp Hochreiter, the mastermind behind LSTM networks and founder of NXAI, shares insights from his journey in AI. He discusses the potential of XLSTM for robotics and industrial simulation. Hochreiter critiques Large Language Models' shortcomings in true reasoning and creativity. He emphasizes the need for hybrid approaches that integrate symbolic reasoning with neural networks. His reflections on the evolution of neural architectures reveal the exciting advancements in memory management and processing efficiency, hinting at a transformative future for AI.

01:07:01

Creator website

Episode guests

Sepp Hochreiter

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Sepp Hochreiter argues that current large language models (LLMs) serve more as advanced databases than true AI due to their lack of reasoning capabilities.

The introduction of XLSTM showcases significant advancements in LSTM technology, enhancing memory management and performance for applications beyond language tasks.

A hybrid approach integrating symbolic and sub-symbolic AI systems is suggested as a potential solution to improve reasoning and knowledge representation in current models.

Deep dives

Limitations of Large Language Models

Large language models (LLMs) are described as advanced database technologies rather than true artificial intelligence. They gather and synthesize existing human knowledge from texts and codes without possessing the ability to generate genuinely new ideas or concepts. The discussion highlights that while LLMs can perform generalizations and combinations of known data, their capabilities are constrained to what they have previously learned. Consequently, they lack the capacity for real reasoning or the generation of innovative code that has not been seen before.

Intro

3min

Exploring the Potential and Limitations of Large Language Models

2min

Exploring Reasoning Limitations in Programming and AI Models

2min

Human vs AI Reasoning: A Deep Dive

4min

Rethinking AI Reasoning and the Rise of LSTM

2min

A Journey Through Neural Networks and Charismatic Leadership

2min

Advancements in Neural Network Architectures

27min

Integrating Symbolic and Neural Approaches in AI

17min

Revolutionizing Simulation with AI

7min

Sepp Hochreiter, the inventor of LSTM (Long Short-Term Memory) networks – a foundational technology in AI. Sepp discusses his journey, the origins of LSTM, and why he believes his latest work, XLSTM, could be the next big thing in AI, particularly for applications like robotics and industrial simulation. He also shares his controversial perspective on Large Language Models (LLMs) and why reasoning is a critical missing piece in current AI systems.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting!

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich.

Goto https://tufalabs.ai/

***

TRANSCRIPT AND BACKGROUND READING:

https://www.dropbox.com/scl/fi/n1vzm79t3uuss8xyinxzo/SEPPH.pdf?rlkey=fp7gwaopjk17uyvgjxekxrh5v&dl=0

Prof. Sepp Hochreiter

https://www.nx-ai.com/

https://x.com/hochreitersepp

https://scholar.google.at/citations?user=tvUH3WMAAAAJ&hl=en

TOC:

1. LLM Evolution and Reasoning Capabilities

[00:00:00] 1.1 LLM Capabilities and Limitations Debate

[00:03:16] 1.2 Program Generation and Reasoning in AI Systems

[00:06:30] 1.3 Human vs AI Reasoning Comparison

[00:09:59] 1.4 New Research Initiatives and Hybrid Approaches

2. LSTM Technical Architecture

[00:13:18] 2.1 LSTM Development History and Technical Background

[00:20:38] 2.2 LSTM vs RNN Architecture and Computational Complexity

[00:25:10] 2.3 xLSTM Architecture and Flash Attention Comparison

[00:30:51] 2.4 Evolution of Gating Mechanisms from Sigmoid to Exponential

3. Industrial Applications and Neuro-Symbolic AI

[00:40:35] 3.1 Industrial Applications and Fixed Memory Advantages

[00:42:31] 3.2 Neuro-Symbolic Integration and Pi AI Project

[00:46:00] 3.3 Integration of Symbolic and Neural AI Approaches

[00:51:29] 3.4 Evolution of AI Paradigms and System Thinking

[00:54:55] 3.5 AI Reasoning and Human Intelligence Comparison

[00:58:12] 3.6 NXAI Company and Industrial AI Applications

REFS:

[00:00:15] Seminal LSTM paper establishing Hochreiter's expertise (Hochreiter & Schmidhuber)

https://direct.mit.edu/neco/article-abstract/9/8/1735/6109/Long-Short-Term-Memory

[00:04:20] Kolmogorov complexity and program composition limitations (Kolmogorov)

https://link.springer.com/article/10.1007/BF02478259

[00:07:10] Limitations of LLM mathematical reasoning and symbolic integration (Various Authors)

https://www.arxiv.org/pdf/2502.03671

[00:09:05] AlphaGo’s Move 37 demonstrating creative AI (Google DeepMind)

https://deepmind.google/research/breakthroughs/alphago/

[00:10:15] New AI research lab in Zurich for fundamental LLM research (Benjamin Crouzier)

https://tufalabs.ai

[00:19:40] Introduction of xLSTM with exponential gating (Beck, Hochreiter, et al.)

https://arxiv.org/abs/2405.04517

[00:22:55] FlashAttention: fast & memory-efficient attention (Tri Dao et al.)

https://arxiv.org/abs/2205.14135

[00:31:00] Historical use of sigmoid/tanh activation in 1990s (James A. McCaffrey)

https://visualstudiomagazine.com/articles/2015/06/01/alternative-activation-functions.aspx

[00:36:10] Mamba 2 state space model architecture (Albert Gu et al.)

https://arxiv.org/abs/2312.00752

[00:46:00] Austria’s Pi AI project integrating symbolic & neural AI (Hochreiter et al.)

https://www.jku.at/en/institute-of-machine-learning/research/projects/

[00:48:10] Neuro-symbolic integration challenges in language models (Diego Calanzone et al.)

https://openreview.net/forum?id=7PGluppo4k

[00:49:30] JKU Linz’s historical and neuro-symbolic research (Sepp Hochreiter)

https://www.jku.at/en/news-events/news/detail/news/bilaterale-ki-projekt-unter-leitung-der-jku-erhaelt-fwf-cluster-of-excellence/

YT: https://www.youtube.com/watch?v=8u2pW2zZLCs

<truncated, see show notes/YT>

Remember Everything You Learn from Podcasts

Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.

Machine Learning Street Talk (MLST)

Sepp Hochreiter - LSTM: The Comeback Story?

Episode guests

Podcast summary created with Snipd AI

Quick takeaways

Deep dives

Limitations of Large Language Models

The Need for New Directions in AI

Challenges in Reasoning Capabilities of AI

Advancements in LSTM and XLSTM Technologies

Integrating Symbolic and Sub-Symbolic Approaches

Remember Everything You Learn from Podcasts