How to choose a model tier without guessing

By John M. · Updated 15 August 2026 · 7 min read

The same job — 6,000 tokens in, 900 tokens out, 200,000 times a month — costs $11,400 on OpenAI GPT-5.6 Sol and $218.40 on DeepSeek V4 Flash. Fifty-two times the money for the same nominal unit of work. No other decision in an AI product has a 52× lever attached to it, and almost nobody makes it with evidence. They make it by reputation, by whatever the founder used in the prototype, or by whichever model was at the top of a leaderboard the week the feature was scoped.

That is a strange way to spend $134,000 a year.

The leaderboard is not your workload

Public benchmarks measure aggregate performance across tasks chosen by other people to be discriminating across models in general. Your feature is one task, with one input distribution, one output format, and one definition of failure. A model that is three points better on a reasoning suite may be worse at emitting the JSON schema your parser expects, and a model that is mediocre in aggregate may be entirely adequate at classifying support tickets into eleven categories.

The counter-argument is that leaderboards correlate with real performance, and they do — loosely, across many tasks, in expectation. But you are not running many tasks in expectation. You are running one, and the variance around that correlation is exactly the thing that decides whether you can drop from $11,400 to $810 a month. Correlation is not a substitute for measurement when the measurement takes an afternoon.

Vibes are worse. "It felt smarter" is a report about a handful of prompts, chosen by someone who already had a hypothesis, scored against no written standard, with no record of what was tried. It is not repeatable, so when a new model lands next month you cannot re-run it.

The method

The whole exercise is five steps and it should take one engineer less than a day.

1. Pull 50 real examples from production traffic. Not 50 examples you wrote. Real inputs, with their real messiness — the truncated ones, the ones in the wrong language, the ones with a pasted email signature at the bottom. Sample them deliberately: roughly 30 typical cases drawn at random, 15 hard cases from the tail of your length or complexity distribution, and 5 known past failures that generated a support ticket or a bug report. If you have no production traffic yet, this method does not apply to you and you should pick the cheap model, ship, and come back in a month with real data.

2. Write pass/fail for each example before you run anything. Binary, and written down. Not a 1–5 score. "Correct category and valid JSON" passes; anything else fails. If an example needs a rubric longer than two sentences, your task definition is the problem, not the model. Writing the criteria before you see any outputs is what stops you from grading generously when the cheap model nearly gets it.

3. Set the bar as a number, in advance. 47 out of 50 — 94% — is a common place to land for a classification or extraction task. Interactive assistance can often live at 42. Anything that writes to a database or sends something to a customer should be higher, or should be gated by a human. Decide before you have a favourite.

4. Sort the candidate models by blended cost on your workload shape, cheapest first. Not by headline input price. On the 6,000-in/900-out shape, Gemini 2.5 Flash at $0.30/$2.50 costs $810 a month and Mistral Large 3 at $0.50/$1.50 costs $870, so Flash goes first. Invert the shape to 1,000 in and 2,000 out and the ordering flips: Mistral Large 3 becomes 200M × $0.50 = $100 plus 400M × $1.50 = $600, so $700, while Gemini 2.5 Flash becomes $60 plus 400M × $2.50 = $1,000, so $1,060. The ladder is a property of your token mix, not of the price list. Use the LLM API cost calculator to build it.

5. Run the ladder from the bottom and stop at the first model that clears the bar. Do not run the whole ladder and pick the best score. The best score is always the most expensive model and you already knew that. The question you are answering is "what is the cheapest thing that works", and the answer is the first rung that passes.

Fifty examples is small, and it is worth being explicit about what that buys you. A model scoring 47/50 has a 95% confidence interval of roughly 83% to 99% on true pass rate — wide enough that a model at 47/50 and a model at 45/50 are not reliably distinguishable. What settles it is more examples, and the practical rule is that 50 is enough to reject a model that is clearly failing and not enough to adjudicate between two that both pass. When two rungs both clear the bar, take the cheaper one and expand the set to 200 before you promote anything to the tier above.

The spread you are trading against

Here is what the ladder costs on that 6,000-in/900-out, 200,000-request workload. Monthly totals are 1,200 million input tokens and 180 million output tokens.

GPT-5.6 Sol costs 2.71× Claude Sonnet 5, which costs 5.19× Gemini 2.5 Flash, which costs 3.71× DeepSeek V4 Flash. Annualised, the top of the ladder is $136,800 and the bottom is $2,620.80 — a difference of $134,179.20 a year on one feature.

Those multiples are the price of your uncertainty. If the eval set says Gemini 2.5 Flash passes at 48/50 and Sonnet 5 passes at 49/50, you are being asked to pay $3,390 a month for one example out of fifty, which is inside the noise. If Flash passes at 31/50, the $3,390 is buying something real and you should pay it without complaint. The eval set is what tells you which of those two situations you are in, and nothing else does.

Cascading: the rung between rungs

The ladder assumes one model handles everything. Often the truth is that 80% of your traffic is easy and the rest is not, and buying the expensive tier for all of it is buying insurance against a minority of requests.

Route it. Send everything to Gemini 2.5 Flash first, detect the cases it handles badly, and escalate those to Claude Sonnet 5. Say 18% escalate. The cheap leg is all 200,000 requests at $810. The escalated leg is 36,000 requests: 36,000 × 6,000 = 216 million input tokens at $2.00 per million is $432.00, and 36,000 × 900 = 32.4 million output tokens at $10.00 per million is $324.00, so $756.00. Combined, $1,566.00 a month against $4,200.00 for all-Sonnet — a 62.7% cut while the hard 18% still gets the expensive model.

The catch is the router. If detection requires a second model to judge the first, you have added a call to every request. Price it before you assume it is prohibitive: judging a 900-token output against a 200-token rubric on Gemini 2.5 Flash-Lite at $0.10/$0.40 costs 200,000 × 1,100 = 220 million input tokens at $0.10 per million, so $22.00, plus 200,000 × 20 = 4 million output tokens at $0.40 per million, so $1.60. Twenty-three dollars sixty against a $1,566 bill. The judge is not the problem.

Latency is. A cascade doubles the wall-clock time for the escalated fraction, because the second call cannot start until the first has finished and been judged. For a nightly job that is free. For a chat assistant, 18% of users waiting twice as long may cost more in churn than the $2,634 you saved. Where a free signal exists — schema validation failure, a refusal, a low retrieval score, an output below a length threshold, an explicit "I don't know" — use it instead of a judge and get the routing for nothing.

Cascades also need re-tuning. Escalation rates drift as your traffic changes, and a router calibrated on last quarter's inputs will quietly escalate 40% by the end of this one. Instrument the rate and alarm on it.

Re-run it when the ground moves

The reason to keep the eval set in version control rather than in a notebook is that the ladder changes underneath you. New models land; prices get cut; a tier you rejected in March passes in September because the vendor shipped a better checkpoint under the same name. Every re-run costs you 50 requests per candidate — for the model above, 50 × 6,000 = 300,000 input tokens and 50 × 900 = 45,000 output tokens, which on GPT-5.6 Sol is $1.50 plus $1.35, so $2.85 to test the most expensive model on the list.

Two dollars eighty-five. Set a calendar reminder for the first Monday of each quarter, check the price change log, re-run the ladder, and when it moves, size the switch with the model switch savings calculator before you write a migration ticket.

All figures in this guide come from the price index, last verified 15 August 2026. Every entry links to the vendor page it was read from — see the price index and the methodology.