How GPU Access Helps AI Startups Be Agile

23 snips

Oct 23, 2024

Explore the critical GPU shortages facing AI startups as large firms dominate access. Discover how a new initiative offers these startups cost-effective solutions for securing GPU resources. Learn about the shift from over-provisioning to demand-based frameworks, enabling agility in infrastructure. Delve into the role of open-source models in enhancing efficiency while reducing costs. Finally, understand the disconnect in AI regulation, highlighting the disparities between training expenses and actual performance outcomes.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

Anthropic's Funding Needs

Anjney Midha recalls getting a call from Anthropic founders needing $500 million for their seed round.
This highlighted the capital-intensive nature of AI startups and their reliance on GPUs.

INSIGHT

Hyperscaler Prioritization

Hyperscalers prioritized long-term contracts during the GPU supply crunch, disadvantaging startups.
Startups faced committing more capital than raised for multi-year contracts.

ADVICE

Capacity Planning for Startups

Avoid long-term GPU commitments as a startup due to capital constraints and unpredictable inference needs.
Start with short-term capacity, assess customer demand, then inform purchasing decisions.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

In this episode of AI + a16z, General Partner Anjney Midha explains the forces that lead to GPU shortages and price spikes, and how the firm mitigates these concerns for portfolio companies by supplying them with the GPUs they need through a program called Oxygen. The TL;DR version of the problem is that competition for GPU access favors large incumbents who can afford to outbid startups and commit to long contracts; when startups do buy or rent in bulk, they can be stuck with lots of GPUs and — absent training runs or ample customer demand for inference workloads — nothing to do with them.

Here is an excerpt of Anjney explaining how training versus inference workloads affect what level of resources a company needs at any given time:

"It comes down to whether the customer that's using them . . . has a use that can really optimize the efficiency of those chips. As an example, if you happen to be an image model company or a video model company and you put a long-term contract on H100s this year, and you trained and put out a really good model and a product that a lot of people want to use, even though you're not training on the best and latest cluster next year, that's OK. Because you can essentially swap out your training workloads for your inference workloads on those H100s.

"The H100s are actually incredibly powerful chips that you can run really good inference workloads on. So as long as you have customers who want to run inference of your model on your infrastructure, then you can just redirect that capacity to them and then buy new [Nvidia] Blackwells for your training runs.

"Who it becomes really tricky for is people who bought a bunch, don't have demand from their customers for inference, and therefore are stuck doing training runs on that last-generation hardware. That's a tough place to be."

Learn more:

Navigating the High Cost of GPU Compute

Chasing Silicon: The Race for GPUs

Remaking the UI for AI

Follow on X:

Anjney Midha

Derrick Harris

Check out everything a16z is doing with artificial intelligence here, including articles, projects, and more podcasts.