What is the Cost of Maintaining the Correctness of a GenAI Service? (268)

Nov 25, 2025

The discussion dives into the costs of maintaining correctness in generative AI services. It highlights how correctness impacts AI economics and product strategies. Jeff contrasts modern AI data centers with traditional ones, exploring architectural implications for cost. You'll hear about the trade-offs between compute costs for self-hosted models versus APIs. The importance of required accuracy levels varies significantly across applications, along with considerations of human involvement and ongoing maintenance costs. The complexities of evolving correctness over time are also examined.

Ask episode

AI Snips

Chapters

Books

Transcript

Episode notes

INSIGHT

Correctness Drives AI Economics

Generative AI changes core economics because maintaining correctness becomes central to costs and strategy.
Correctness can shift a product from high-margin software to low-margin service-like economics.

INSIGHT

Foundation Models Add A New Cost Layer

Foundation models add a fourth architectural component: app, database, compute, and model.
That extra component fundamentally alters cost structure and competitive dynamics.

ADVICE

Account For Initial And Ongoing Compute Costs

Calculate compute costs including energy, cooling, and cloud provider bills for both initial build and ongoing inference.
Choose between self-hosting a downloaded model or using cloud APIs based on upfront versus recurring cost tradeoffs.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

This week’s podcast is about estimating the costs of providing GenAI products and services. And this is changing.

You can listen to this podcast here, which has the slides and graphics mentioned. Also available at iTunes and Google Podcasts.

Here is the link to the TechMoat Consulting.

Here is the link to our Tech Tours.

My approach to assessing GenAI operating costs is the below checklist.

What is the cost of compute, including energy and cooling? Initial cost vs. ongoing?
1. The core compute is going to drive a lot of the costs. Especially if you are using an AI cloud service provider. If you have a downloaded open-source model, then it’s mostly the initial cost.
2. These compute costs dependent on the requirements of the AI workloads:
  1. The compute requirements
  2. The timing requirements
  3. The memory requirements
What is the cost of creating and maintaining the desired correctness over time?
1. How accurate and correct you need the foundation model to be is a big deal. And this factor determines how much of the operations are done by software versus humans. It can make the costs of GenAI products look a lot more like services than traditional software.
2. My main questions are:
  1. How much does correctness matter? What level of correctness does the product need to compete? What is the cost of inaccuracy?
  2. How does correctness change over time? Is it stable and flat lining, advancing, naturally changing or deteriorating?
  3. How much of a long tail is there in the domain? Is the long tail a liability or a benefit?
  4. How much of the process is iterative? How much are humans in the loop?
  5. What is the initial cost of training and getting to the desired level of correctness?
  6. What is the ongoing cost of inference at the desired level of correctness? How much are humans in the loop?
  7. What is the cost and frequency of fine tuning and/or retraining?
What are the cost implications of increasing scale?
1. What are the scale advantages vs. disadvantages?
2. Does the Jevon paradox apply?

--------

I am a consultant and keynote speaker on how to accelerate growth with improving customer experiences (CX) and digital moats.

I am a partner at TechMoat Consulting, a consulting firm specialized in how to increase growth with improved customer experiences (CX), personalization and other types of customer value. Get in touch here.

I am also author of the Moats and Marathons book series, a framework for building and measuring competitive advantages in digital businesses.

This content (articles, podcasts, website info) is not investment, legal or tax advice. The information and opinions from me and any guests may be incorrect. The numbers and information may be wrong. The views expressed may no longer be relevant or accurate. This is not investment advice. Investing is risky. Do your own research.

Support the show