3288: MLPerf vs Moore’s Law: Redefining AI Progress

May 24, 2025

David Kanter, Founder and Executive Director of MLCommons, delves into the world of AI performance benchmarking. He shares how MLPerf, the industry's leading benchmark, addresses the question of what constitutes 'good' AI as it evolves beyond Moore’s Law. Kanter discusses the importance of transparency and collaboration in developing standards, while also tackling the challenges of energy consumption in AI. He highlights innovative solutions such as low-precision computation and the role of community collaboration in driving AI advancements.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

Serendipitous Start to Leadership

David Kanter described how he became the MLCommons founder by showing up at a meeting and becoming the group's scribe.
This serendipitous start led him from analyst to leader of a global AI benchmarking effort.

ADVICE

Use MLPerf for Directional Comparisons

Use MLPerf benchmarks as directional indicators when evaluating AI hardware performance.
They offer a practical, representative way to compare solutions closely aligned with real workloads.

INSIGHT

AI Progress Outruns Moore's Law

AI performance is improving about 50 times faster than Moore's Law due to advances beyond just hardware.
Innovations in software, algorithms, and system scale drive this explosive growth in AI capability.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

What happens when the world's most powerful AI systems are measured by the same yardstick?

In this episode of Tech Talks Daily, I spoke with David Kanter, Founder and Executive Director of MLCommons, the organization behind MLPerf, the industry's most recognized benchmark for AI performance. As AI continues to outpace Moore’s Law, businesses and governments alike are asking the same question: how do we know what “good” AI performance really looks like? That’s exactly the challenge MLCommons set out to address.

David shares the story of how a simple suggestion at a Stanford meeting led him from analyst to the architect of a global benchmarking initiative. He explains how MLPerf benchmarks are helping enterprises and policymakers make informed decisions about AI systems, and why transparency, neutrality, and open collaboration are central to the mission.

We explore what’s really driving AI’s explosive growth. It’s not just about chips. Smarter software, algorithmic breakthroughs, and increasingly scalable system designs are all contributing to performance improvements far beyond what Moore’s Law predicted.

But AI’s rapid progress comes with a cost. Power consumption is quickly becoming one of the biggest challenges in the industry. David explains how MLCommons is helping address this with MLPerf Power and why infrastructure innovations like low-precision computation, advanced cooling, and even proximity to power generation are gaining traction.

We also talk about the decision by some major vendors not to participate in MLPerf. David offers perspective on what that means for buyers and why benchmark transparency should be part of any enterprise AI procurement conversation.

Beyond the data center, MLCommons is now benchmarking AI performance on consumer hardware through MLPerf Client and is working on domain-specific efforts such as MLPerf Automotive. As AI shows up in smartphones, vehicles, and smart devices, the need for clear, fair, and relevant performance measurement is only growing.

So how do we measure AI that is everywhere? What should buyers demand from vendors? And how can the industry ensure that AI systems are fast, efficient, and accountable? Let’s find out.