“OpenAI finetuning metrics: What is going on with the loss curves?” by jorio, James Chua

Nov 26, 2025

Dive into the intricacies of OpenAI's fine-tuning metrics as experts decode the hidden complexities behind loss and accuracy calculations. Discover the curious case of two extra tokens that impact these metrics, shrouded in sparse documentation. Follow their journey through controlled experiments, where a focused analysis on color datasets reveals surprising results. Learn how batch size influences accuracy fluctuations and the broader implications this has for GPT-4.1's performance. A fascinating exploration for anyone intrigued by AI training nuances!

Ask episode

AI Snips

Chapters

Transcript

Episode notes

INSIGHT

Metrics Include Two Hidden Tokens

OpenAI's fine-tuning loss and accuracy use standard cross-entropy and token-level accuracy with teacher forcing.
Both metrics include two extra tokens beyond the visible assistant response, likely EOS plus another special token.

INSIGHT

Extra Tokens May Appear Internally

The two extra tokens are not necessarily appended at the end of the visible assistant reply.
If those tokens are predicted with near certainty they still affect reported loss and accuracy calculations.

ANECDOTE

Controlled Fine-Tuning Experiments

The authors fine-tuned GPT‑4.1 on ~6,000 conversations per dataset to test their hypothesis.
They used datasets of 2, 3, 5, and 6 single-token color labels and trained for two epochs to convergence.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Introduction

For our current project, we've been using the OpenAI fine-tuning API. To run some of our experiments, we needed to understand exactly how the reported metrics (loss and accuracy) are calculated. Unfortunately, the official documentation is sparse, and the most detailed explanation we could find was the following table from Microsoft's Azure documentation:

Our experimental results didn't match what we expected from these definitions. So we ran controlled experiments to reverse-engineer the metrics.

What we found:

The loss and accuracy metrics are indeed based on standard cross-entropy loss and token-level accuracy with teacher forcing, but with a critical caveat: both metrics include two additional tokens beyond the visible assistant response. These are likely an end-of-sequence (EOS) token plus another special token, though this is not mentioned in the documentation.

Some calculations

To be concrete, suppose that you are performing SFT on the following conversation:

User: blah blah blah
Assistant: TOKEN1

where TOKEN1 is a single token. We claim that there are two additional tokens, TOKEN2 and TOKEN3, such that the loss is

<span>_text{Loss} = - frac{1}{3} left( log p(text{TOKEN1}) + log p(text{TOKEN2}) + log p(text{TOKEN3}) right)_</span>

and the accuracy is

<span>_text{ACC} = frac{text{NUMBER OF CORRECTLY [...]

---

Outline:

(00:13) Introduction

(01:21) Some calculations

(02:14) Experiments

(03:40) Results

(05:28) Conclusions

(06:12) Acknowledgments

The original text contained 1 footnote which was omitted from this narration.

---

First published:
November 24th, 2025