How to A B test Claude models on cost.

The reason teams keep paying top tier rates for work a cheaper model would do is not stubbornness, it is fear of an untested switch. Nobody wants to move a production workload from Opus to Sonnet, or Sonnet to Haiku, on a hunch and then discover quality slipped where it mattered. The answer is not to avoid the switch, it is to test it properly. A well run A B test turns a risky guess into a measured decision, because it shows you the exact cost saving and the exact quality difference side by side on your own traffic. This guide lays out how to run that test so the result is trustworthy enough to act on, and defensible enough that finance, engineering, and the contract all benefit from it.

Test cost and quality together, never separately

The single most common mistake is measuring one dimension at a time. A test that only tracks cost tells you the cheaper model is cheaper, which you already knew, and nothing about whether it is good enough. A test that only tracks quality tells you the models differ without telling you whether the difference is worth the money. The A B test has to capture both on the same requests, so that for each candidate you can state the cost per call and the quality against your bar in one view. Only then can you make the real decision, which is not whether the cheaper model is worse, it is whether it is worse in a way that matters for this task at the price difference on offer. Often it is not worse in any way you can measure, and the saving is free.

Define the quality bar before you run anything

An A B test is only as good as its quality metric, so define the bar before you start. For a classifier that means an accuracy threshold against a labeled set. For a generation task it means a rubric a reviewer or an evaluation model can apply consistently: accuracy, completeness, tone, format, and the specific error types you will not tolerate. Write this down before you see results, because a bar invented after the fact bends to whatever the test produced. With a fixed bar, the test gives a clean verdict: the candidate either clears it at lower cost, in which case you switch, or it falls short, in which case you keep the incumbent and you know exactly why. The bar is what makes the result evidence rather than opinion.

Run both models on the same real traffic

The test must run on representative traffic, ideally a sample of real production requests rather than a handful of cherry picked examples. Send the same inputs to both the incumbent and the candidate, capture both outputs, the token counts, and the cost for each, and score both against the bar. Running on real traffic matters because the edge cases that break a cheaper model live in the long tail, not in the clean examples, and a test on tidy inputs will overstate how well the candidate does. A shadow run, where the candidate processes copies of live requests without serving users, is the safest way to gather this at scale, because it exposes the candidate to real conditions with zero user risk.

Make the sample big enough to trust

A switch worth millions over a contract term deserves more than a dozen test cases. The sample needs to be large enough that the quality difference you measure is real and not noise, and large enough to surface the rare failures that matter. How large depends on how varied your traffic is and how costly a miss would be, but the principle is to keep gathering until the result is stable, the measured quality gap stops moving as you add more cases, and the cost saving is clear. A test that is too small produces a confident wrong answer, which is worse than no test, because it gets acted on. When the numbers stop moving, you have enough.

Read the result as a commercial decision

When the test is done you have, for each task, the cost per call on each model and the quality against the bar. Now the decision is commercial. If the candidate clears the bar at materially lower cost, switch, and bank the saving. If it clears the bar but only saves a little, weigh the saving against the switching effort. If it misses the bar, keep the incumbent, but record by how much, because a near miss today often becomes a clear pass after the next model update, and you will want the baseline. Run this across your largest workloads and the cumulative effect is the forty to seventy percent aggregate saving that model routing delivers, captured one tested, defensible switch at a time rather than in one nervous leap.

The test as a negotiation asset

The evidence an A B test produces is worth more than the immediate saving. When you commit to Anthropic, you commit against a baseline, and a baseline backed by tested model choices is one you can defend line by line. A buyer who can show that each workload runs on the cheapest model that clears a documented bar walks into the commitment conversation with a lower, credible forecast and the standing that comes with rigor. The teams that overcommit are the ones that never tested, ran everything on the top tier out of caution, and locked the inflated run rate into a multi year deal. Test first, route to the result, then commit, and the same quality costs less on the bill and in the contract.

Run the test, then run the deal

• Measure cost and quality together on the same requests, never one at a time.
• Fix the quality bar in writing before you see any results.
• Use real production traffic, ideally a shadow run, so the long tail is in the sample.
• Keep gathering until the quality gap stabilizes, then you have enough.
• Read the result as a commercial decision and record the near misses for next time.
• Carry the tested baseline into the Anthropic commitment so you commit to a truer number.

We run these tests for clients and then take the optimized baseline straight into the Anthropic negotiation, so the saving shows up twice, on the monthly bill and in the commitment you sign. If you want a fixed fee or gainshare engagement that pairs the testing with the contract work, get a quote below, and read the full method in our token optimization playbook.