AI's Unsung Hero: Data Labeling and Expert Evals

28 snips

Jun 27, 2025

Manu Sharma, CEO of Labelbox and an expert in data labeling for AI, joins to discuss the evolution of data labeling from supervised learning to advanced reinforcement learning. He highlights how the shift to foundation models and generative AI has transformed the industry, emphasizing the emerging role of 'aligners'—top professionals who ensure high-quality training data. Manu also touches on the competitive landscape of AI, particularly the impact of recent acquisitions like Meta's purchase of Scale AI, underscoring the growing importance of data and talent in the AGI race.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

Labelbox's Early Launch Story

Manu Sharma shared how Labelbox began by addressing data labeling challenges in computer vision for geospatial data.
They launched on Reddit with a collaborative web-based labeling tool that quickly gained customers across diverse sectors.

INSIGHT

Same Core Problem, New Context

The core problem Labelbox solves remains managing expert-driven high-quality data labeling.
Today's data production combines human experts and AI software to improve model capabilities in complex domains.

INSIGHT

Evolution of Human Supervision

Human input in AI training has become more sophisticated, focusing on preference and quality judgment.
Experts now guide AI models in assessing the quality and nuances of answers, not just correctness.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Labelbox CEO Manu Sharma joins a16z Infra partner Matt Bornstein to explore the evolution of data labeling and evaluation in AI — from early supervised learning to today’s sophisticated reinforcement learning loops.

Manu recounts Labelbox’s origins in computer vision, and then how the shift to foundation models and generative AI changed the game. The value moved from pre-training to post-training and, today, models are trained not just to answer questions, but to assess the quality of their own responses. Labelbox has responded by building a global network of “aligners” — top professionals from fields like coding, healthcare, and customer service, who label and evaluate data used to fine-tune AI systems.

The conversation also touches on Meta’s acquisition of Scale AI, underscoring how critical data and talent have become in the AGI race.

Here's a sample of Manu explaining how Labelbox was able to transition from one era of AI to another:

It took us some time to really understand like that the world is shifting from building AI models to renting AI intelligence. A vast number of enterprises around the world are no longer building their own models; they're actually renting base intelligence and adding on top of it to make that work for their company. And that was a very big shift.

But then the even bigger opportunity was the hyperscalers and the AI labs that are spending billions of dollars of capital developing these models and data sets. We really ought to go and figure out and innovate for them. For us, it was a big shift from the DNA perspective because Labelbox was built with a hardcore software-tools mindset. Our go-to market, engineering, and product and design teams operated like software companies.

But I think the hardest part for many of us, at that time, was to just make the decision that we're going just go try it and do it. And nothing is better than that: "Let's just go build an MVP and see what happens."

Follow everyone on X:

Manu Sharma

Matt Bornstein

Check out everything a16z is doing with artificial intelligence here, including articles, projects, and more podcasts.