The hidden costs of AI that never reach your price card

By John M. · Updated 15 August 2026 · 7 min read

The dashboard said $1,120. That was one customer-support summarisation feature's spend on GPT-5.6 Terra last month: 100,000 calls at roughly 2,000 input and 600 output tokens each, which at $2.00 and $12.00 per million tokens comes to 200M x $2.00/M = $400 of input and 60M x $12.00/M = $720 of output. The same feature cost the business about $12,400 to operate. Neither number is wrong. Only one of them appears on a price card.

What follows is the rest of the ledger, roughly in the order teams discover it, with every line sized against that $1,120 so you can see which ones deserve attention and which are rounding error.

Retries: 3% on a normal month, 20% on a bad day

Every production client retries on 429s and 5xx responses, usually three attempts with exponential backoff. On a healthy month that is cheap. If 4% of calls need one retry and 0.6% need two, you are billed for 104,600 calls instead of 100,000, which adds about $35.

The bad day is the one to budget for. A partial generation that hits your client timeout at 55 seconds has already produced most of its output tokens, and you are billed for them even though your application discarded the response. Stack a three-attempt retry policy on top and one eight-hour provider degradation touching 10% of the month's traffic adds 10,000 x 2 extra attempts x $0.0112 = $224 in a single day, a fifth of the entire month's bill. Worse, aggressive retries during a degradation are how a slow provider becomes an outage: every client in your fleet triples its request rate at exactly the moment the provider has least headroom.

Cap total attempts, add jitter, and set your timeout below the point where a retry can still land inside the user's patience.

Failed generations, where the multiplier lives

Text is forgiving because a mediocre answer is still usable. Images and video are not: a generation is either shipped or thrown away, and the thrown-away ones are billed identically.

Imagen 4 costs $0.04 per image. At a one-in-four acceptance rate, which is generous for a first pass against a brand brief, each image you actually ship costs 4 x $0.04 = $0.16. Producing 40 usable catalogue images means 160 generations and $6.40, not $1.60.

Video compounds the same arithmetic against a much larger base. Veo 3.1 at $0.40 per second makes an eight-second clip cost $3.20; at one acceptable clip in three, the delivered clip cost $9.60. Kling 2.5 Turbo at $0.042 per second puts the same eight seconds at $0.336, so a one-in-three rate still lands at $1.01. The honest number is worse again, because early prompt iteration on a new brief routinely runs eight or ten attempts before the first keeper, and only later settles into a steady acceptance rate.

Budget generative media on delivered assets, never on generations. The image and video cost calculator has an acceptance-rate field for exactly this reason; if you leave it at 100% you are forecasting a fantasy.

Evaluation runs, which can outspend production

During a build, evals are the production workload. A 300-case eval set run against four prompt variants and three candidate models is 3,600 calls per sweep. On Claude Sonnet 5 at 3,000 input and 700 output tokens, that is 10.8M x $2.00/M = $21.60 plus 2.52M x $10.00/M = $25.20, so $46.80.

Then you add an LLM judge, because human scoring does not survive contact with a daily sweep. A judge call on Claude Opus 5 carrying the input, the output and the rubric at 4,000 input and 300 output tokens costs 14.4M x $5.00/M = $72.00 plus 1.08M x $25.00/M = $27.00, a further $99.00. Judging costs more than twice what it judges. Sixty sweeps across a six-week build is roughly $8,760, spent in a period when production spend was zero.

After launch the trimmed regression set does not go away. Eight sweeps a month of a 120-case set is about $400, which is 36% of that $1,120 API bill, forever.

Observability, priced in gigabytes

You cannot debug a model you did not record, so serious deployments store the full prompt and completion for every call. At 2,600 tokens per call, payloads plus metadata run about 10 KB, so 100,000 calls a month is roughly 1 GB of trace data before you keep a single day of history beyond thirty.

Whether that costs $50 or $300 depends on your vendor's rate card, and we are not going to invent a figure. Do the multiplication yourself: calls x payload size x retention, then read it off the card. The decision that matters is not the price but the sampling rate, because teams that sample at 5% to save money reliably discover that the one call they need to inspect is in the 95%.

There is a second-order cost here that surprises people. Trace payloads are verbatim customer text, so the moment you store them you have created a new store of personal data with a retention policy, a deletion path and a redaction step that someone has to build and someone else has to audit. That work is measured in engineer-weeks, not dollars per gigabyte, and it lands on whoever shipped the feature.

Guardrails, which double your request count and barely touch your bill

Screening input and output through a classifier doubles or triples the number of requests your system makes, which matters for rate limits and latency. It barely registers on cost, provided you pick the right model. Screening 2,000 input tokens and 600 output tokens on Gemini 2.5 Flash-Lite at $0.10/$0.40 costs about $0.000268 per protected call, or $27 a month across 100,000 calls. That is 2% on top of the API bill for a meaningful reduction in the tail risk below.

Do it with the same tier as the main call and the picture changes: pushing 2,600 tokens per call through GPT-5.6 Terra as a screener costs $0.00544 each and $544 a month, adding 49%. Moderation is a job for the cheapest model that passes your own eval set.

Human review, usually the largest single line

This is where the money is, and it never appears on any AI vendor's page. A reviewer costing $28 an hour fully loaded who clears 40 items an hour costs $0.70 per reviewed item. The model call costs $0.0112. Review one item in ten and you add 10,000 x $0.70 = $7,000 a month, six times the entire API spend.

Two consequences follow. First, the review rate is the single most valuable number in the system, so the work that lowers it, better routing, confidence thresholds, letting the model abstain, is worth far more than any model price negotiation. Second, moving from GPT-5.6 Terra to Luna at $0.20/$1.20 would cut the API line by 90%, saving about $1,000, and would be an unambiguous loss if it pushed the review rate from 10% to 12%.

Keeping the thing working while models retire

Prompts are calibrated against a specific model's behaviour. When that model is deprecated, the calibration is gone: few-shot examples that steered the old model do nothing for the new one, output formats drift, and the eval set has to be rerun end to end. Budget fifteen engineer-days per forced migration, which at a fully loaded $900 a day is $13,500. Two deprecations a year is $27,000, or $2,250 a month against an API bill of $1,120.

This line is the reason cheap-model migrations often fail to pay. Saving $1,000 a month while adding a re-qualification cycle you would not otherwise have run is not a saving.

The cost of being wrong

The last line has no invoice at all. If 2% of 100,000 monthly responses are wrong in a way a customer notices, and one in twenty of those produces a support contact costing $12, that is 2,000 x 0.05 x $12 = $1,200 a month in rework. Credits, churn and the occasional regulatory conversation sit on top and refuse to be averaged.

The ledger

Every line there is derived from the arithmetic above except trace storage, which carries a placeholder until you read it off your vendor's card, and the two labour lines, which use your costs rather than anyone's list price. The API line is 9% of the total. That ratio is not universal, but the shape is: in any system with a human in the loop, labour dominates, and in any system without one, the cost of being wrong dominates instead.

Rebuild this table with your own numbers before you approve a budget. The AI ROI calculator takes the labour side, and if generative media is in scope, run the acceptance rate through the image and video cost calculator first. If the business case only survives when the API line is the only line, it does not survive.

All figures in this guide come from the price index, last verified 15 August 2026. Every entry links to the vendor page it was read from — see the price index and the methodology.