DisTrO and the Quest for Community-Trained AI Models

Sep 27, 2024

Guest

Jeffrey Quesnelle

Guest

Bowen Peng

Bowen Peng and Jeffrey Quesnelle from Nous Research discuss their mission to revive open-source AI, emphasizing the DisTrO project, which enables rapid training of AI models over the internet. They explore the challenges faced by independent builders in AI and the critical role of community collaboration. The conversation dives into impressive innovations like the Hermes models, designed for neutral interactions and enhanced with synthetic data. They reflect on the tension between decentralization and centralization in AI protocols and advocate for community-driven solutions.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

Jeff's Career Path

Jeffrey Quesnelle worked in automotive for 15 years, then transitioned to AI after discovering Stable Diffusion.
He joined Nous Research after connecting with like-minded individuals on Reddit.

ANECDOTE

Bowen's AI Journey

Bowen Peng was inspired to study machine learning after seeing generative models in his first programming class.
He joined Nous Research after connecting with Jeffrey Quesnelle on Reddit.

INSIGHT

Nous Research's Focus

Nous Research focuses on fundamental AI research, pushing boundaries with minimal compute.
Their Hermes models prioritize user intent over strict guardrails, allowing diverse personas.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

In this episode of AI + a16z, Bowen Peng and Jeffrey Quesnelle of Nous Research join a16z General Partner Anjney Midha to discuss their mission to keep open source AI research alive and activate the community of independent builders. The focus is on a recent project called DisTrO, which demonstrates it's possible to train AI models across the public internet much faster than previously thought possible. However, Nous is behind a number of other successful open source AI projects, including the popular Hermes family of "neutral" and guardrail-free language models.

Here's an excerpt of Jeffrey explaining how DisTrO was inspired by the possibility that major open source AI providers could turn their efforts back inward:

"What if we don't get Llama 4? That's like an actual existential threat because the closed providers will continue to get better and we would be dead in the water, in a sense.

"So we asked, 'Is there any real reason we can't make Llama 4 ourselves?' And there is a real reason, which is that we don't have 20,000 H100s. . . . God willing and the creek don't rise, maybe we will one day, but we don't have that right now.

"So we said, 'But what do we have?' We have a giant activated community who's passionate about wanting to do this and would be willing to contribute their GPUs, their power, to it, if only they could . . . but we don't have the ability to activate that willingness into actual action. . . . The only way people are connected is over the internet, and so anything that isn't sharing over the internet is not gonna work.

"And so that was the initial premise: What if we don't get Llama 4? And then, what do we have that we could use to create Llama 4? And, if we can't, what are the technical problems that, if only we slayed that one technical problem, the dam of our community can now flow and actually solve the problem?"

Learn more:

DisTrO paper

Nous Research

Nous Research GitHub

Follow everyone on X:

Bowen Peng

Jeffrey Quesnelle

Anjney Midha

Check out everything a16z is doing with artificial intelligence here, including articles, projects, and more podcasts.