Evaluating Fine-Tuned Models: The Two-Suite Rule

Evaluating a fine-tuned model means running two eval suites, not one. A task-specific harness proves the fine-tune improved on the behavior you targeted. A regression harness checks that the base model's general capabilities did not quietly degrade while you were watching the first number. Skip the second suite and you ship a model that scores well on your demo set and gets measurably worse at everything else.

I have watched this exact failure happen: a rubric score that looked great, and a support queue that got worse two weeks later. The fine-tune had learned the tone the team wanted. It had also lost the ability to format an order number correctly, a capability nobody thought to test because the fine-tune was "about tone." That gap, between a passed eval and a shipped regression, is why this article exists.

Key takeaways

• Run two eval suites, not one. A task-specific harness proves the fine-tune improved on its target behavior. A regression harness proves nothing else broke.
• Build the golden eval set before training, not after. A held-out set carved out post-hoc risks leaking through the training data it's supposed to check against.
• Catastrophic forgetting is real and non-obvious. One empirical study found it across models from 1B to 7B parameters, with larger models in that range sometimes forgetting more, not less.
• LLM-as-a-judge rewards fluency over correctness. A fine-tune that sounds more confident can out-score a more accurate base model on a judge's rubric.
• Own your harness. Vendor eval products get deprecated. Build the scoring logic to run against any model, not just the one API hosting it today.

What "evaluating a fine-tuned model" actually means

Evaluating a fine-tuned model is not the same exercise as a general LLM evaluation. A general eval asks whether a model is good at a broad range of tasks. Evaluating a fine-tune asks a narrower, harder question: did this specific training run make the model better at the thing you trained it for, without making it worse at everything else it used to do.

That second half is the part teams skip. It's tempting to treat a fine-tune like a feature you ship and measure in isolation: run the new model against your target task, compare the score to the old one, ship if it's higher. That comparison answers only half the question. A fine-tune sits on top of a base model with general capabilities you're already relying on elsewhere in your product. This is what "the human evaluates" looks like for a fine-tune: not one reviewer skimming outputs, but two suites run every time.

Build the golden eval set before you fine-tune, not after

Your eval set has one job: tell you the truth about a model it has never seen. That only works if you carve it out before training starts, not after. A fine-tuning run typically pulls from the same pool of production examples you'd naturally reach for when assembling a held-out set later, so by the time training finishes the line between "data the model trained on" and "data you're testing against" has usually blurred.

Picture a team that fine-tunes a support classifier, then goes looking for held-out examples once training wraps. Half the "held-out" set turns out to be near-duplicates of training examples pulled from the same queue, the same week. The eval passes, because the model has effectively seen the questions before, just phrased slightly differently. Nothing in that number tells you how the model handles a genuinely new case.

The fix is mechanical, not clever. Before you touch a training script, split your data: a training set, a validation set for hyperparameters, and a golden set the model never sees until the final evaluation run. Stratify it across the categories and edge cases that matter in production, not just the easy majority class. Then leave it alone. Edit the golden set after a disappointing score and you no longer have a golden set, you have a number you've started managing instead of measuring. I walk through the full dataset discipline, held-out splits through the kill decision, in Fine-Tune or Don't.

Task-specific evals: proving the fine-tune did its job

The task-specific suite answers one question: is the fine-tuned model better than the base model at the exact behavior you trained it for. For anything with a checkable right answer, score it with exact-match, not a vibe check. A classification fine-tune gets graded on precision and recall against the golden set. A format-compliance fine-tune gets graded on whether the output parses against your schema, full stop, no partial credit for "close."

For softer targets, tone, helpfulness, adherence to a style guide, write a rubric before you write a single example, and score against it with a human or a well-scoped judge, not an open-ended "rate this 1 to 10." A rubric with three or four checkable criteria produces a score you can audit later. A 1-to-10 rating produces a number nobody, including the person who assigned it, can fully explain.

Run the same task-specific suite against the base model, prompted as well as you know how, alongside the fine-tune. That comparison is the whole point. A fine-tune that beats a lazily-prompted base model proves nothing. A fine-tune that beats your best prompt-engineering effort on the base model is evidence worth trusting.

Regression evals: catching catastrophic forgetting against the base model

Catastrophic forgetting is what happens when a fine-tune reshapes a model toward your training distribution and quietly loses capability it never got retrained on. It's measurable and more common than intuition suggests. An empirical study of continual fine-tuning found forgetting across models ranging from 1B to 7B parameters, and, counterintuitively, larger models in that range forgot more, not less (Luo et al., 2023). Decoder-only models retained more than encoder-decoder models under the same regime, one more reason architecture belongs in the eval conversation.

The regression suite exists to catch exactly this. Run the general-capability benchmark suite you'd use to evaluate any base model against the fine-tune, the same way you ran it before training, and compare the two score sets side by side. Any category that drops is a regression, whether or not anyone thought to mention it in the fine-tuning ticket.

Method matters too. Parameter-efficient approaches like LoRA touch a smaller share of a model's weights than a full fine-tune, part of why they tend to forget less: they simply learn less. I go deeper on that trade-off in my comparison of LoRA and full fine-tuning. Neither method makes the regression suite optional. It changes how much regression to expect, not whether you need to look.

LLM-as-a-judge for open-ended outputs, and where it lies to you

For open-ended generation, where there's no single correct string to match, LLM-as-a-judge is the practical way to score at scale. It's good, not great, and the gap between those two words is where a bad ship decision hides. GPT-4 as a judge reached over 80% agreement with human preferences on MT-Bench, roughly matching typical agreement between two human raters (Zheng et al., 2023). The same paper documents the failure modes: position bias, verbosity bias, and self-enhancement bias, plus limited reasoning ability on harder judgments.

Verbosity bias is the one that bites hardest when grading a fine-tune. A fine-tuned model that has learned to sound more confident, structured, and thorough will out-score a terser, more accurate base model on a judge's rubric, because the judging model is itself prone to rewarding fluency. Picture a fine-tune that picked up the habit of restating the question and adding a summary line before its answer. The judge rates it higher every time, not because it's more correct, but because it reads like a better answer. General-purpose judges caught outright hallucinations at only 53.8% to 58.5% accuracy in one practitioner's independent testing, well below what most teams assume when they reach for "just ask GPT-4 to grade it" (Yan, Task-Specific LLM Evals).

That's not an argument against using a judge, only against using it alone for a fine-tune whose whole job is correctness. Use it to triage volume and catch obvious regressions; reserve rubric-scored, exact-match evals on the golden set as the metric that gates a release.

Tooling: what to actually run the harness with

You have three options, not mutually exclusive. An open-source harness like lighteval or the EleutherAI evaluation framework runs standard benchmarks against any model you can serve, a reasonable default for the regression suite. A hosted eval product is fast to start with if you already live inside that provider's ecosystem. A custom harness, a script that loads your golden set and scores both models against your rubric, is what you build for the task-specific suite, because nobody else has your rubric.

Be careful how much of your ship process depends on a single vendor's eval surface. As of 2026, OpenAI's own Evals API is being sunset: it goes read-only on October 31, 2026, and shuts down entirely on November 30, 2026, with OpenAI directing teams toward its newer Datasets product instead (OpenAI, Evals guide). That's not a knock on OpenAI. It's the concrete argument for owning your eval harness: the scoring logic, the golden set, and the comparison pipeline should run independent of any one provider's API, portable to whatever model you're evaluating next year.

The specific library matters less than the property this snippet illustrates: your harness calls a model, not a vendor's eval endpoint. Swap the model, swap the provider, and the scoring logic doesn't move.

Wiring the harness into your ship process

An eval harness that runs once, produces a report, and gets filed away isn't an eval harness. It's a slide. The version that protects you runs as a gate in your deployment pipeline: every candidate fine-tune runs the full task-specific and regression suite before it's allowed to replace the model in production, the same way a test suite blocks a bad code deploy.

Set an explicit threshold before you look at the numbers, not after. Decide in advance how much task-specific improvement justifies the run, and how much regression you'll tolerate, if any. Then let the CI gate enforce it. A model that beats the baseline on your target task but regresses on a capability your product depends on should fail automatically, not get a judgment call from whoever's in the room when the numbers land.

This also means re-running the regression suite on a schedule, not only at fine-tune time. A provider issuing a silent model update, drift in production traffic, or a prompt change upstream can all shift the ground your fine-tune sits on. Treat the harness the way you treat a test suite: something that runs continuously, not something you run once and trust forever.

When to trust the fine-tune vs. roll back to base model + RAG

The decision isn't "fine-tune vs. RAG" in the abstract. It's what your own eval numbers told you. Use the two suites as a decision table, not a report you write up after the fact.

That last row is the one teams resist, because a fine-tuning project already has a training run and a sunk-cost narrative behind it by the time the numbers come in. But if the task-specific suite shows no real win and the regression suite shows any cost, the fine-tune has made the model worse for no offsetting benefit. Kill it, and go back to the base model with better retrieval for the gap you were trying to close. My framework for when to fine-tune in the first place is the mirror image of this decision: a weak case going in makes a middling eval result confirmation, not a surprise.

How do I know if my fine-tuned model is actually better than the base model?

Run both models against the same task-specific golden set and the same general-capability regression suite. "Better" means a real improvement on the task-specific suite with no meaningful regression on the general suite, not a higher score on a single metric you happened to check.

What is catastrophic forgetting, and how do I test for it before I ship?

Catastrophic forgetting is when a fine-tune reshapes a model toward its training distribution and quietly loses capability it was never retrained on. Test for it by running your standard general-capability benchmark suite against the fine-tune the same way you ran it against the base model, and comparing the two score sets side by side before you ship.

Can I use GPT-4 or Claude to grade my fine-tuned model's outputs?

Yes, for triage and for scoring at scale on open-ended outputs, but not as your only gate. LLM judges carry documented biases toward fluency and verbosity, which can make a confident, well-formatted fine-tune out-score a more accurate base model. Pair the judge with exact-match or rubric scoring on held-out examples for anything where correctness is the point.

How big does my eval set need to be before I can trust the results?

Large enough to cover the categories and edge cases that matter in production, stratified rather than random. A few hundred well-chosen, held-out examples per category tell you more than a few thousand pulled loosely from the same distribution as your training data.

If you're past deciding whether to fine-tune and need the eval harness built alongside the training run, not bolted on after a bad launch, that's the kind of build hiring a ViitorCloud ML developer is for. Two eval suites, run before every ship, is the difference between a fine-tune you can defend and one you're hoping nobody stress-tests.