The real cost of self-hosting an open-weight model

By John M. · Updated 15 August 2026 · 8 min read

A RunPod H100 SXM bills $3.29 an hour. Over an average month of 730 hours that is $2,401.70 — call it $2,402 — and you pay all of it whether the GPU spends the month saturated or idling at 4% while your users are asleep. Token APIs bill for tokens. Rented GPUs bill for time. Every argument about self-hosting is downstream of that one difference, and most self-hosting business cases quietly assume a utilisation figure nobody has measured.

The deployment

Two replicas, each a single H100 SXM running a 70B open-weight model at FP8. Weights at one byte per parameter come to roughly 70GB, which fits in an 80GB card with about 10GB left for KV cache and activations — tight, and the tightness is why your concurrency ceiling is lower than you hoped. Assume each replica sustains 900 output tokens per second in aggregate under continuous batching. That is a reasonable figure for this class of setup, and it is also the single number most worth verifying with a load test before you commit, because everything below scales linearly with it.

Fixed cost: 2 × $2,401.70 = $4,803.40 a month.

Capacity: 1,800 output tokens per second across both replicas, over 730 × 3,600 = 2,628,000 seconds a month. Take the same workload shape used elsewhere on this site — 6,000 input tokens, 900 output tokens per request — and one replica turns out one request's worth of output per second, so the pair does 2 requests per second, or 5,256,000 requests a month flat out. (Prefill is assumed absorbed within that throughput figure. On a 6,000-token input it is not free, and a load test is the only way to find out how much of your decode budget it eats.)

At 25% utilisation you serve 1,314,000 requests a month and still pay $4,803.40.

The alternative is Llama 3.3 70B on Together at $1.04 per million tokens in each direction. Per request that is (6,000 + 900) × $1.04 ÷ 1,000,000 = $0.0071760. At 1,314,000 requests, $9,429.26 a month.

So the rented hardware is $4,625.86 a month cheaper at 25% utilisation. Self-hosting wins, on the numbers as usually presented. That is the naive comparison, and it is where most business cases stop.

Break-even on hardware alone is $4,803.40 ÷ $0.0071760 = 669,371 requests a month, which is 12.7% of capacity. Anything above roughly an eighth of what the cards can do and the GPUs look cheaper.

What the naive comparison leaves out

Someone has to run it. Not build it — run it. Version upgrades to the inference server, driver and CUDA pinning, OOM crashes when a user pastes a 40,000-token document, autoscaling that fights itself, a load balancer, metrics, an incident when a host disappears. Charge that at 0.3 of one engineer, and at a fully loaded $15,000 a month that is $4,500. Total self-hosted cost becomes $9,303.40 against Together's $9,429.26 — a difference of $125.86, or 1.3%, in self-hosting's favour. The entire $4,625.86 raw advantage is consumed by less than a third of one person.

Break-even with that charge included is $9,303.40 ÷ $0.0071760 = 1,296,601 requests, or 24.7% of capacity. Which is to say: at the 25% utilisation this deployment was specified at, the two options are a dead heat, and one of them requires you to hire.

High utilisation and low latency are opposing goals. The table above treats 75% utilisation as a target, and for a queue-driven batch job it is one. For interactive traffic it is not. Queueing theory is unsentimental about this: as a server approaches saturation, waiting time rises non-linearly, so a fleet averaging 75% will produce tail latencies that a fleet averaging 25% never sees. You cannot simultaneously claim the cost advantage of a busy GPU and the latency profile of an idle one. Pick which of the two your feature needs, and size for that.

The input side is doing more work than the throughput figure admits. The 900 tokens per second above is a decode number, and decode is what people benchmark. But this workload sends 6,000 input tokens for every 900 it generates — nearly seven to one — and prefill on that is real compute that competes with decode for the same card. A hosted API charges input and output separately and transparently; a rented GPU charges you neither, and simply gets slower. If your prompts are long, measure end-to-end request throughput at your real prompt length, not tokens per second on a short one.

On-call is not included in the 0.3 FTE. A serving stack that fronts a customer-facing feature needs someone reachable at 03:00. If you have an existing rota, the marginal cost is small. If you do not, you are creating one for a saving that has already been shown to be zero at this volume.

Model evaluation is now your job, repeatedly. Every open-weight release invites a re-evaluation, and each one means benchmarking on your own eval set, checking throughput on your hardware, and a migration if it wins. Managed APIs give you the same choice with a string change. The method for doing this properly is in choosing a model tier; the point here is that self-hosting means paying for it in engineering time as well as tokens.

Quantisation is a quality decision you have taken by accident. FP8 on a 70B is close to lossless in aggregate, but the loss is not evenly distributed — it concentrates in the long tail, on the exact hard cases your eval set exists to catch. Your throughput assumption depends on that quantisation, so the performance number and the quality number are tied together, and you cannot change one without re-measuring the other.

Cold starts destroy the scale-to-zero fantasy. Turning replicas off overnight is the obvious answer to a utilisation problem, until you time it: pulling and loading 70GB of weights takes minutes, so the first request after a scale-up waits. Autoscaling a GPU fleet is not autoscaling a web service.

Capacity planning is the utilisation problem, restated. 25% utilisation is rarely a choice. It is the arithmetic consequence of sizing for peak and living at the mean. Business-hours traffic with a 4× peak-to-mean ratio gives you exactly this: you buy peak capacity, you pay for it continuously, and you run at a quarter of it. APIs invert that relationship — you pay the mean and the vendor absorbs the peak. That inversion is most of what you are buying.

The vendor absorbs hardware refresh. RunPod lists B200 at $6.79 an hour against $3.29 for H100 SXM. The moment a B200 delivers more than 2.06× an H100's throughput on your model, your H100 fleet is the expensive option and migrating is your problem. The same chip also spans an extraordinary price range across suppliers — $3.29 at RunPod, $3.99 at Lambda ($2,912.70 a month), $6.155 per GPU at CoreWeave ($4,493.15 a month), and from $1.60 an hour on the Vast.ai marketplace ($1,168 a month per replica, which would take hardware-only break-even down to 6.2% of capacity). Chasing that spread is real money and also a real, recurring procurement job, on heterogeneous hosts with varying reliability and data-handling postures.

And the competitor is not the model you chose. The sharpest problem with the whole exercise is that you are comparing your GPUs against Llama 3.3 70B on Together, when the actual alternative is the cheapest hosted model that passes your evals. Llama 4 Scout via Fireworks at $0.15/$0.60 costs (6,000 × $0.15 + 900 × $0.60) ÷ 1,000,000 = $0.00144 per request, so $1,892.16 a month at 1,314,000 requests. Qwen3 235B via Together at $0.20/$0.60 comes to $0.00174 per request, or $2,286.36. Both are less than half the hardware-only cost of the deployment above, before any engineering charge at all. If either passes your eval set, the self-hosting case is not close.

Where self-hosting genuinely wins

Sustained high volume. The economics invert once the cards are busy. At 75% utilisation — 3,942,000 requests a month — Together costs $28,287.79 while the fleet costs $4,803.40 plus $4,500 of engineering, so $9,303.40. That is $18,984.39 a month, or $227,812.68 a year, which funds a small team and leaves change. This is the real case, and it has a precondition: sustained, not peak. If your 3,942,000 requests arrive in six hours a day, you are back to buying peak capacity.

Data residency and control. No per-token price beats a regulator. If data cannot leave a jurisdiction or a VPC, the comparison is not a comparison. Confirm the constraint first, though — most major vendors now offer regional endpoints, zero-retention terms, and signed data agreements, and a surprising number of self-hosting projects are launched to satisfy a requirement that a contract clause would have met.

Fine-tuned models nobody else will host. A LoRA on an open-weight base that no serverless provider carries is a genuine reason to own the serving stack. Weigh it with the fine-tuning vs prompting calculator before you assume the fine-tune is necessary, but where it is, hosting it yourself is often the only option.

Predictable batch workloads. This is the case that dissolves the utilisation problem entirely, because you control the arrival rate. A nightly job that must finish by morning can saturate the cards for six hours and release them, and it can run on interruptible marketplace capacity — Vast.ai A100 80GB at $0.52 an hour is $379.60 for a full month, and a fraction of that if you only rent overnight. A job that can be restarted from a checkpoint does not care about pre-emption, which is exactly the property spot capacity demands.

Before you provision anything, measure three numbers: your actual peak-to-mean traffic ratio over a full week, the output tokens per second your model really achieves on the card you intend to rent, and the price of the cheapest hosted model that clears your eval bar. Then put them into the self-hosting vs API break-even calculator. If the answer is not obvious by a factor of two, it is not obvious at all, and the API keeps the engineering time.

All figures in this guide come from the price index, last verified 15 August 2026. Every entry links to the vendor page it was read from — see the price index and the methodology.