Inference versus training cost differences

This week’s podcast is about estimating the costs of providing GenAI products and services. And this is changing.

You can listen to this podcast here, which has the slides and graphics mentioned. Also available at iTunes and Google Podcasts.

Here is the link to the TechMoat Consulting.

Here is the link to our Tech Tours.

My approach to assessing GenAI operating costs is the below checklist.

1. What is the cost of compute, including energy and cooling? Initial cost vs. ongoing?
   1. The core compute is going to drive a lot of the costs. Especially if you are using an AI cloud service provider. If you have a downloaded open-source model, then it’s mostly the initial cost.
   2. These compute costs dependent on the requirements of the AI workloads:
      1. The compute requirements
      2. The timing requirements
      3. The memory requirements
2. What is the cost of creating and maintaining the desired correctness over time?
   1. How accurate and correct you need the foundation model to be is a big deal. And this factor determines how much of the operations are done by software versus humans. It can make the costs of GenAI products look a lot more like services than traditional software.
   2. My main questions are:
      1. How much does correctness matter? What level of correctness does the product need to compete? What is the cost of inaccuracy?
      2. How does correctness change over time? Is it stable and flat lining, advancing, naturally changing or deteriorating?
      3. How much of a long tail is there in the domain? Is the long tail a liability or a benefit?
      4. How much of the process is iterative? How much are humans in the loop?
      5. What is the initial cost of training and getting to the desired level of correctness?
      6. What is the ongoing cost of inference at the desired level of correctness? How much are humans in the loop?
      7. What is the cost and frequency of fine tuning and/or retraining?
3. What are the cost implications of increasing scale?
   1. What are the scale advantages vs. disadvantages?
   2. Does the Jevon paradox apply?

 --------

I am a consultant and keynote speaker on how to accelerate growth with improving customer experiences (CX) and digital moats.

I am a partner at TechMoat Consulting, a consulting firm specialized in how to increase growth with improved customer experiences (CX), personalization and other types of customer value. Get in touch here.

I am also author of the Moats and Marathons book series, a framework for building and measuring competitive advantages in digital businesses.

This content (articles, podcasts, website info) is not investment, legal or tax advice. The information and opinions from me and any guests may be incorrect. The numbers and information may be wrong. The views expressed may no longer be relevant or accurate. This is not investment advice. Investing is risky. Do your own research.

Support the show