Is self-hosting an open-weight model cheaper than using an API?

Only at high, sustained utilisation. An API bills per token and costs nothing when idle. Self-hosting bills GPU hours continuously whether or not requests arrive, so the economics invert: APIs favour spiky or low-volume workloads, self-hosting favours steady high throughput that keeps the hardware busy. Before comparing, add the costs that are not the GPU — engineering time, model updates, evaluation, scaling, monitoring and on-call. Teams routinely compare a GPU hourly rate against an API token rate and omit everything else.

FinOps for AI · Cost architecture · August 2026

Fine-Tune, Prompt, RAG or Self-Host? The AI Cost Decision

Q: Is fine-tuning cheaper than prompting?

Sometimes, and it depends almost entirely on prompt length and call volume. Fine-tuning moves instructions out of the prompt and into the model's weights, so you stop paying for those input tokens on every single call. If you send a long system prompt or many few-shot examples millions of times, that saving compounds. But fine-tuning adds a training cost, often a higher per-token serving rate, and a re-training cost every time the task changes — so at low volume, or on a task that shifts often, prompting stays cheaper.

Q: When is RAG cheaper than fine-tuning?

When the knowledge changes. RAG retrieves facts at query time, so updating what the model knows means updating an index rather than re-training. Fine-tuning bakes behaviour into weights, so every knowledge change means another training run. The rule of thumb: fine-tune to change how the model behaves — its format, tone, or task structure — and use RAG to change what it knows. RAG costs more per call because retrieved context inflates input tokens, plus vector store and embedding costs, but it avoids repeated training and stale answers.

Q: How do I work out the break-even?

Compare cost shapes, not headline rates. Write out each option as fixed cost plus marginal cost per request: prompting is nearly all marginal; fine-tuning is a fixed training cost plus a marginal serving cost; self-hosting is largely fixed. Then plot them against your realistic monthly request volume — the crossover point is your break-even. Do it with measured token counts from your own traffic rather than estimates, because the input-to-output ratio drives the answer and it is rarely what teams assume.

By the CloudFinOpsKit team. 9 min read.

These four options do not just cost different amounts — they cost in different shapes. One is almost entirely marginal, one is fixed, two are a mix. That is why "which is cheaper?" has no general answer, and why teams that pick based on a headline rate so often end up surprised. Here is how each option actually bills, what genuinely drives the break-even, and the costs each one hides.

The four cost shapes

Option	Fixed cost	Marginal cost per request	Idle cost
Prompting	None	High — every instruction re-billed each call	Zero
RAG	Low — index & embeddings	High — retrieved context inflates input	Low — vector store
Fine-tuning	Training run, repeated on change	Lower input, often higher rate	Zero to low
Self-hosting	Very high — GPU capacity	Near zero	Full — GPUs bill regardless

Read that table as a slope-and-intercept problem. Prompting starts at zero and climbs steeply. Self-hosting starts high and stays nearly flat. Fine-tuning and RAG sit between. The right answer is wherever your realistic volume falls relative to the crossovers — which is why the same decision goes the other way for two teams with identical use cases and different traffic.

Prompting: highest marginal cost, zero commitment

You pay for every token, every time — including the system prompt and any few-shot examples, re-billed on each call. A 900-token system prompt sent two million times a month is 1.8 billion input tokens spent restating instructions.

It wins when volume is low or unpredictable, the task is still changing, or you are still discovering what "good" looks like. Its real advantage is optionality: nothing is sunk, and you can change the behaviour this afternoon.

Before optimising anything else, fix prompting waste — output ceilings, trimmed context, and prompt caching where the same prefix repeats. Those cost nothing to implement and lower the baseline every other option is measured against. There is no sense computing a fine-tuning break-even against a bloated prompt.

RAG: pay to look things up

RAG adds retrieval infrastructure — embeddings, a vector store, the query-time lookup — and then increases input tokens, because retrieved context is prepended to the prompt. Its cost lands in two places, and teams often budget only the first.

It wins when knowledge changes. Facts live in an index you can update in minutes, without touching the model. That is a cost advantage as much as an accuracy one: the alternative to re-indexing is re-training.

The most common RAG cost mistake is retrieving too much — twenty chunks where five would answer the question — which inflates the expensive part of every call. Retrieval quality is a cost lever, not just a relevance one.

Fine-tuning: move instructions into the weights

Fine-tuning's cost logic is simple: stop paying for the same instructions on every call. If behaviour that currently takes a long prompt can be baked into the model, those input tokens disappear from every future request.

Against that, three costs:

The training run — one-off, but repeated every time the task materially changes.
Serving rate — fine-tuned models often bill at a higher per-token rate, and on some platforms carry a hosting charge while deployed.
Re-training on drift — the cost that surprises people. A tuned model is a snapshot; when requirements move, you pay again.

It wins when the prompt is long, the task is stable, and volume is high enough that saved input tokens exceed training plus the rate premium. Stability matters as much as volume — high volume on a task that changes monthly means paying for training repeatedly.

The clean split: fine-tune to change how the model behaves; use RAG to change what it knows. Teams that fine-tune to teach facts end up re-training every time the facts move.

Self-hosting open weights: the shape inverts

Here the economics flip. An API costs nothing when idle; GPUs bill continuously whether or not a request arrives. You are no longer buying tokens, you are buying capacity — and capacity you do not use is pure waste, exactly like an over-provisioned reservation.

So self-hosting only pays at high, sustained utilisation. Spiky traffic is its worst case: you size for the peak and pay for it around the clock.

And the GPU rate is not the cost. The comparison teams actually need includes engineering time to deploy and maintain, model and dependency updates, evaluation harnesses, autoscaling, monitoring, and on-call. Comparing a GPU hourly rate to an API token rate — and nothing else — is the single most common error in this decision.

How to actually decide

Fix the prompt first. Output ceilings, tighter context, caching. Every subsequent comparison is against this baseline.
Measure your real numbers — input and output tokens per request, and requests per month, from your own traffic. The input-to-output ratio drives the answer and is rarely what people assume.
Write each option as fixed + marginal. Not a headline rate — the two numbers that define the line.
Plot against realistic volume, including the pessimistic case. Find the crossover.
Include change frequency. How often does this task shift? That determines how many training runs, or re-indexes, you are really buying.
Divide by successful outputs, not calls. An option that is cheaper per call but needs more retries is not cheaper — see cost per successful output.

The reasonable default

For most teams: prompt first, add RAG when knowledge needs to be current, fine-tune when a long prompt has stabilised at high volume, and self-host only when utilisation is genuinely sustained and you have the operational appetite.

Move down that list when the numbers say so — not because the next step sounds more sophisticated. Each step trades flexibility for a lower marginal cost, and that trade only pays once the workload has stopped changing.

Know your baseline before you commit

Every option above is priced relative to what you spend today — and most teams cannot say what that is per model, let alone per request. The CloudFinOpsKit FinOps Agent reads it from your real billed cost across Azure OpenAI, Amazon Bedrock and Vertex AI, flags the waste that would distort any comparison — output bloat, cold prompt caches, zombie deployments, reserved capacity you are not using — and an AI analyst tells you what to fix first. It runs read-only in your own environment. Get the baseline honest, then run the break-even.

FAQ

Is fine-tuning cheaper than prompting?

It depends on prompt length, call volume and task stability. Fine-tuning removes repeated instruction tokens from every call, but adds training cost, often a higher serving rate, and re-training whenever the task changes. At low volume or on a shifting task, prompting stays cheaper.

When is RAG cheaper than fine-tuning?

When the knowledge changes. RAG updates an index; fine-tuning needs another training run. Fine-tune to change how the model behaves, use RAG to change what it knows.

Is self-hosting cheaper than an API?

Only at high sustained utilisation. APIs bill per token and cost nothing idle; GPUs bill continuously. And the GPU rate is not the cost — add engineering, updates, evaluation, scaling, monitoring and on-call.

How do I work out the break-even?

Express each option as fixed cost plus marginal cost per request, then plot against realistic monthly volume using measured token counts from your own traffic. The crossover is your answer.