When is Vertex AI Provisioned Throughput cheaper than pay-as-you-go?

Provisioned Throughput buys reserved generative capacity for a fixed fee, so it only pays when your sustained utilisation is high enough that the equivalent pay-as-you-go token spend would exceed the commitment. It is a bad fit for spiky or exploratory workloads, where you pay for a peak you rarely reach. Measure your real sustained token throughput over a full billing period first, then compare that to the committed cost — and re-check after any prompt or model change, because both move token volume.

What is context caching in Vertex AI and when does it save money?

Context caching lets you reuse a large, stable block of input context across many requests instead of resending and re-paying for it every call. Cached input tokens bill at a lower rate than standard input tokens, with a storage charge for keeping the cache alive. It pays off when the same substantial context — a document, a system prompt, a schema — is reused frequently within the cache lifetime. It does not help one-off prompts or workloads where the context changes every request.

Google Cloud · FinOps for AI · August 2026

Vertex AI Cost Optimization: Token FinOps for Gemini on Google Cloud

Q: How do I see what Vertex AI is actually costing me?

The Cloud Billing console shows Vertex AI as a service total, which is rarely enough to act on. For per-model detail, enable the Cloud Billing export to BigQuery and group by SKU — Gemini input tokens, output tokens, and cached input tokens are billed as separate SKUs, so a single query tells you which model and which token direction is driving the spend. Pair that with the Vertex AI token-count metrics in Cloud Monitoring to get requests and tokens per model alongside the dollar figure.

Q: Does Vertex AI charge for idle endpoints?

Yes, for dedicated infrastructure. A Vertex AI endpoint with a deployed model reserves machines and bills for the replicas while they are up, whether or not you send a single prediction — that is the classic zombie cost in Vertex. Gemini API calls priced per token do not bill when idle, but Provisioned Throughput does, because you are buying reserved capacity. Undeploy models from endpoints you are not serving, and set minimum replicas to zero where the deployment supports scale-to-zero.

By the CloudFinOpsKit team. 9 min read.

Vertex AI spend arrives as one line on the bill, and almost none of it tells you what to change. The bill says "Vertex AI: $8,400". It does not say that one endpoint has served zero predictions in three weeks, or that a verbose system prompt is adding 40% to every single call. This is how to get from the service total to the four things that actually move Vertex and Gemini cost — and how to tell which of them applies to you.

First, get per-model visibility (the service total is useless)

You cannot optimise what you cannot attribute. Two data sources between them give you everything:

Cloud Billing export to BigQuery — the authoritative dollars. Gemini bills input tokens, output tokens and cached input tokens as separate SKUs, so grouping by SKU immediately shows which model and which token direction is driving spend. This is the number you can defend in a finance review.
Cloud Monitoring — the volume. Vertex publishes token-count and request metrics per model, which gives you the denominator: tokens per request, requests per day, and the input-to-output ratio.

Dollars alone tell you Vertex is expensive. Dollars plus volume tell you why — and "why" is the only thing you can act on. If you have not enabled the BigQuery billing export yet, do that before anything else in this article; without it you are optimising blind.

1. Idle endpoints — the cleanest money on the list

This is the one people miss, because it breaks the mental model. Gemini API calls are priced per token, so they cost nothing when idle. But a Vertex AI endpoint with a deployed model reserves machines and bills for those replicas while they are up — whether or not a single prediction arrives.

The result is a model someone deployed for a demo in March, still quietly billing in August. It has no token cost, so it never shows up in a token analysis. It shows up as machine hours.

What to do: list your endpoints, check prediction counts over the last 30 days in Cloud Monitoring, and undeploy anything at zero. Where the deployment supports it, set minimum replicas to zero so the endpoint scales down instead of idling. Treat "deployed" as a state with a running cost, not a free parking spot.

2. Output bloat — you are paying most for the tokens you generate

Output tokens are consistently the more expensive direction. If your average response is far longer than the answer actually needs, that premium applies to every single call, forever.

The usual causes are boring and very fixable: no max_output_tokens ceiling, a prompt that invites the model to explain its reasoning at length when you only wanted a classification, or a JSON schema the model pads with prose. A summarisation endpoint returning 800 tokens where 200 would do is a permanent 4× on the expensive half of the bill.

What to do: measure average output tokens per request per model, then ask whether the consumer of that output needs all of it. Set explicit output ceilings. For structured tasks, constrain the response format so the model cannot ramble.

3. Context bloat — and when caching actually pays

The mirror problem: oversized input. Retrieval-augmented pipelines are the classic offender — stuffing twenty retrieved chunks into the prompt when five would answer the question, or resending an entire document on every turn of a conversation.

Two different fixes, and choosing the wrong one wastes effort:

If the context is unnecessary — retrieve less. Tighter retrieval, better ranking, fewer chunks. This is free and always right.
If the context is necessary and repeated — use context caching. Cached input tokens bill at a lower rate than standard input, with a storage charge for keeping the cache alive. It pays when the same substantial context is reused frequently within the cache lifetime.

Caching a context you only use once is worse than not caching it — you pay storage for nothing. The deciding question is reuse frequency, not context size.

4. Batch prediction — the discount you get for not being in a hurry

Plenty of AI workloads are not interactive: nightly enrichment, backfills, bulk classification, evaluation runs. If nobody is waiting on the response, online prediction is the wrong pricing model. Batch prediction is billed at a lower rate precisely because it is not latency-sensitive.

What to do: audit which of your Vertex workloads have a human waiting. Anything that runs on a schedule, processes a queue, or backfills history is a batch candidate. This is often the single largest structural saving available, and it changes nothing about output quality.

Provisioned Throughput: measure before you commit

Provisioned Throughput buys reserved generative capacity for a fixed fee. It pays only when sustained utilisation is high enough that equivalent pay-as-you-go token spend would exceed the commitment.

The failure mode is committing on a peak. Teams size to their busiest hour, then run at a fraction of it and pay for headroom they never touch — the AI equivalent of an over-provisioned reservation. Worse, a single prompt or model change can move token volume enough to invalidate the sizing.

What to do: measure real sustained throughput across a full billing period before committing, size to the sustained floor rather than the peak, and re-check after any material prompt or model change.

Do them in this order

Idle endpoints — pure waste, zero risk, immediate.
Batch what isn't interactive — structural, no quality trade-off.
Output ceilings — cheap, and it compounds on every call.
Context: retrieve less, then cache what's genuinely reused.
Provisioned Throughput — only once the workload is stable and measured.

Note the shape: the first three are configuration, not model quality. You can do them without an accuracy conversation, which is why they get done. Commitments come last because they lock in whatever inefficiency is still present.

Make it repeatable

AI cost drifts faster than infrastructure cost, because a prompt change ships in an afternoon and can move token volume by double digits. A one-off cleanup will be stale within a quarter.

The CloudFinOpsKit FinOps Agent runs this analysis over your own project read-only — Vertex AI alongside Azure OpenAI and Amazon Bedrock, and alongside the rest of your cloud estate — pricing every finding from your real billed cost rather than list prices. It flags output-heavy usage, idle deployments and reserved-capacity under-use, then an AI analyst tells you which to fix first and what it is worth. Nothing is deployed into your project and your data never leaves it.

FAQ

How do I see what Vertex AI is actually costing me?

Enable the Cloud Billing export to BigQuery and group by SKU — input, output and cached input tokens bill separately, so one query shows which model and which direction drives the spend. Add Cloud Monitoring's token and request metrics for the volume denominator.

Does Vertex AI charge for idle endpoints?

Yes, for dedicated infrastructure. An endpoint with a deployed model bills for its replicas while they are up, regardless of traffic. Per-token Gemini calls do not bill when idle, but Provisioned Throughput does — you are buying reserved capacity.

When is Provisioned Throughput cheaper than pay-as-you-go?

Only when sustained utilisation is high enough that the equivalent token spend would exceed the commitment. Measure a full billing period first, size to the sustained floor, and re-check after prompt or model changes.

What is context caching and when does it save money?

It reuses a large, stable block of input context across requests at a lower cached-input rate, plus a storage charge. It pays when the same substantial context is reused frequently within the cache lifetime — not for one-off prompts.