Why output tokens cost more than input tokens

By John M. · Updated 15 August 2026 · 6 min read

There is nothing about an output token that makes it intrinsically more valuable than an input token. Both are integers in the same vocabulary. The asymmetry — GPT-5.6 Sol at $5.00 in and $30.00 out, a ratio of 6:1 — is not about the token at all. It is about when, and how, the GPU does the work.

When a request arrives, the model does two mechanically different jobs. The first is prefill: it takes the entire prompt, all 8,000 or 40,000 tokens of it, and pushes it through every layer of the network in one go. Because every token in the prompt is already known, the attention and feed-forward computations for all of them can be expressed as large matrix multiplications and dispatched to the GPU simultaneously. This is exactly the workload a modern accelerator is built for. Arithmetic intensity is high — many floating-point operations per byte fetched from memory — so the chip runs near its compute ceiling. Prefilling 8,000 tokens does not take 8,000 times as long as prefilling one. On an H100 it is a handful of large matmuls, and the marginal cost of the 8,000th prompt token is close to nothing.

Then the model has to write, and the parallelism disappears. Generating token one requires a full forward pass. Token two cannot start until token one exists, because token one is part of its input. Token three waits on token two. Every output token is its own forward pass through the entire network, and each of those passes computes a single token's worth of activations while still having to read every weight in the model out of high-bandwidth memory. For a dense 70-billion-parameter model at 16-bit precision, that is roughly 140GB of weight traffic per token generated. An H100 has about 3.35TB/s of memory bandwidth, which puts a hard ceiling on tokens per second per request that has nothing to do with how much compute the chip has spare. Decode is memory-bandwidth-bound; prefill is compute-bound. The chip is mostly idle during decode, waiting on memory.

The KV cache makes it worse as the response grows. Each generated token attends to every token before it, so the model keeps the key and value vectors for the whole sequence in GPU memory and reads them on every step. That cache grows linearly with sequence length, so the per-step memory traffic rises as the response gets longer — the thousandth output token is more expensive to produce than the tenth. It also consumes memory that would otherwise hold other users' requests. This is the part that shows up on the invoice most directly: a serving cluster's economics depend on how many concurrent requests fit on each GPU, and a long-context, long-output workload evicts its neighbours. Prefill batches beautifully across users; decode batching is capped by KV cache footprint.

Add it up and one output token costs the provider somewhere between three and ten times what one input token costs, depending on model size, precision, batch strategy and how long the responses run. The price cards reflect that spread rather than a single industry constant:

The flat-rate rows are the interesting ones, because they are not evidence that decode is cheap on those models. Ministral 3 and Llama 3.3 70B obey exactly the same physics as Sol. What differs is the pricing model. A host serving open-weight models is reselling GPU time: it rents or owns the hardware — an H100 SXM is $3.29 an hour on RunPod, $3.99 on Together's own cloud — measures aggregate throughput across a mixed workload of long prompts and short prompts and everything in between, divides total cost by total tokens, and quotes one blended number with a margin on top. The prefill-heavy customers subsidise the decode-heavy ones, and the host does not care as long as the mix holds. Frontier labs, running their own models on their own silicon at much larger scale, have both the incentive and the data to price each phase separately.

Model size compounds the effect. Ministral 3's decode penalty is real but small in absolute dollars, because the weight traffic per token scales with parameter count — a few billion parameters is a fraction of the memory reads a frontier model needs per step. Below a certain size the marginal cost of decode is small enough that a single blended rate loses the provider nothing worth the billing complexity, and the $0.10 figure is set by competitive positioning rather than by the cost curve. DeepSeek V4 Flash at $0.14 / $0.28 shows the middle ground: a 2:1 ratio, which is what a sparse mixture-of-experts model with a small active-parameter count per token looks like when someone does price the phases separately.

What this means for what you write

Output is where the money is on any feature that generates more than a few hundred tokens, and the fix is almost always in the prompt rather than in the model choice.

Take a support-reply drafting feature: 1,200 input tokens of ticket context and instructions, 700 output tokens of drafted reply, 250,000 requests a month, running on Claude Opus 5. Input costs 1,200 × 250,000 = 300 million tokens × $5.00 = $1,500. Output costs 700 × 250,000 = 175 million × $25.00 = $4,375. Total $5,875, of which output is 74%. Rewrite the instruction to demand a reply of at most 200 words and the average output drops to around 400 tokens: 100 million × $25.00 = $2,500, and the month lands at $4,000. That is $1,875 saved monthly, $22,500 a year, with the input side untouched and the same model answering.

Preamble is pure waste at the output rate. "Certainly — here is a draft reply based on the ticket details you provided:" plus a closing offer to adjust the tone runs to about 32 tokens that no user reads. Across 250,000 requests that is 8 million tokens, $200 a month on Opus 5, for conversational padding. Telling the model to return the reply body only costs one sentence of system prompt at the input rate.

Cap max_tokens at a value your feature can actually use, and treat the cap as a cost control rather than a safety net. If 1% of those 250,000 requests run away to a 4,096-token limit, that tail is 2,500 × 4,096 = 10.24 million tokens, $256 a month, against the $43.75 those same requests would have cost at the expected 700 tokens. The tail is six times the body.

Prefer structured output. Asking for JSON with three named fields does two things at once: it removes the narrative scaffolding around the answer, and it makes the length predictable enough to cap tightly. A prose answer that averages 700 tokens with a long tail frequently becomes a 220-token object with almost no variance.

Do this arithmetic before you conclude you need a cheaper model. If brevity gets your feature under budget, you keep the better model; if it does not, the model switch savings calculator will show what moving down a tier is worth at your input/output mix, and the LLM API cost calculator will price the combination.

All figures in this guide come from the price index, last verified 15 August 2026. Every entry links to the vendor page it was read from — see the price index and the methodology.