Why doesn't normal cost anomaly detection work for AI spend?

Classic detectors watch dollars per service per day, and AI spend hides three failure modes that dollars alone can't separate: volume (more tokens), mix (traffic moving to a pricier model), and shape (responses getting longer, and output tokens cost 3–5× input tokens). A doubled bill could be growth, a routing bug, or a verbose-output regression — telling them apart requires watching tokens, ratios, and the blended rate, not just cost.

How do I detect that traffic switched to a more expensive AI model?

Track the blended rate: total AI service cost divided by total tokens (in thousands) per day. Model prices differ several-fold, so if the blended $/1K-token rate jumps 25%+ while volume stays flat, the model mix changed — typically a router falling back to a bigger model or a config change. Comparing the trailing 7-day median against the prior 21 days catches it within days.

What is an idle PTU and why is it a cost anomaly?

Provisioned throughput (Azure OpenAI PTUs, Amazon Bedrock Provisioned Throughput) bills a fixed hourly rate for reserved capacity whether you use it or not. A deployment consuming a fraction of its provisioned capacity is pure spend with no matching usage — a standing anomaly that daily-spike detection never flags because the cost is perfectly, expensively flat.

AI & tokenomics · July 2026

Detecting AI Cost Anomalies: Token Spikes, Model-Mix Shifts, and Idle PTUs

By the CloudFinOpsKit team. 10 min read.

An AI workload can double its cost overnight without deploying a single new resource. No new VM, no scaled-up cluster, nothing in the change log — just a retry loop, a prompt edit that made responses twice as long, or a router quietly falling back to the expensive model. Classic cost anomaly detection, which watches dollars per service per day, sees one number move on one line item ("Azure OpenAI" or "Amazon Bedrock") and can't tell you which of those three very different problems you have. Token spend needs its own detectors. Here's why — and the four that work.

Why token spend breaks classic anomaly detection

Traditional cloud cost is mostly provisioned: you pay for what you deployed, so cost moves when infrastructure moves, and a dollar spike maps to a resource change you can find. Token spend is pure consumption, and that breaks the classic detector's assumptions three ways:

Cost has three independent drivers. Spend = volume (tokens) × mix (which models) × shape (how many of those tokens are output — and output tokens cost 3–5× input tokens). A dollar-only detector sees the product, not the factors. A 2× bill could be healthy growth, a routing bug, or a verbosity regression — with three completely different fixes.
The baseline is legitimately volatile. Usage-driven spend swings with traffic, launches, and batch jobs, so an ML model watching dollars either alerts constantly or widens its band until real problems fit inside it.
The worst anomalies are invisible in dollars. A model-mix shift at flat traffic, output bloat at flat request counts, and an idle PTU billing its flat hourly rate all look like "normal" or "slightly up" to a dollar detector. The signal lives in tokens and ratios, not cost.

This is the same lesson as AI cost governance generally: the old playbook doesn't transfer. You have to meter the unit that drives the cost — see token economics for the metering foundation this all builds on.

Detector 1 — tokens/day spikes

The volume detector. Pull daily input and output token counts per AI account — Azure Monitor metrics for Azure OpenAI (ProcessedPromptTokens, GeneratedTokens), CloudWatch for Bedrock (InputTokenCount, OutputTokenCount) — and run spike detection on tokens per day, not dollars: compare each day against the median of its recent same-weekday values, flag when the modified z-score clears 3.5, with an absolute floor (100K tokens/day) so a toy workload tripling from nothing doesn't page anyone.

This catches the volume failures: a prompt/retry loop, an abusive or leaked API key, an unannounced batch job, an agent stuck re-processing the same documents. Because token metrics land in near-real-time while billing data lags a day or more, the token detector fires first — often a full day before the same problem is visible in cost.

Detector 2 — output:input ratio bloat

The shape detector. Track the daily output:input token ratio and compare the trailing 7-day median against the prior 21 days; flag when the ratio rose ≥ 30% with material output volume behind it. A stable workload has a remarkably stable ratio — when it climbs, responses got longer, and since output tokens cost 3–5× input, a ratio drift is a direct money leak at completely flat traffic.

The usual causes: a system-prompt edit that invited verbosity ("explain your reasoning"), a model version change with chattier defaults, a dropped max_tokens cap, or a feature that started returning full documents instead of summaries. Output caps are one of the four levers in the token optimization framework — this detector is what tells you the cap quietly stopped working.

Detector 3 — blended $/1K-token rate shifts

The mix detector. Compute the daily blended rate: AI service cost ÷ (total tokens / 1,000), on days with enough volume to make the number meaningful (≥ 100K tokens). Compare trailing 7-day median vs the prior 21 days and flag a shift of ≥ 25% in either direction.

Model prices differ several-fold, so the blended rate is a fingerprint of your model mix. A rate that jumped at flat volume means traffic moved to a pricier model — a router falling back when the small model erred, a developer "temporarily" pointing at the frontier model, a config change nobody reviewed. A rate that dropped is usually your optimization landing, but confirm it: it can also mean a premium workload silently failed. On AWS, this pairs with per-model CloudWatch visibility — see Bedrock cost optimization for building that view.

Detector 4 — idle provisioned throughput

The standing anomaly. Provisioned throughput — PTUs on Azure OpenAI, Provisioned Throughput on Bedrock — bills a fixed hourly rate for reserved capacity, used or not. That makes it invisible to every detector above: cost is perfectly flat, tokens may be near zero, and nothing "changed." It's not a spike; it's a leak with a subscription.

The check is utilization against capacity: a provisioned deployment consuming a small fraction of what it reserves is paying committed rates for idle capacity, and the fix is to downsize the commitment or move that traffic to pay-as-you-go until utilization justifies provisioning. This is a capacity-utilization check rather than a time-series one, which is why it belongs in your AI workload review as a standing item, not just in the daily anomaly feed.

All four, automated on both clouds. The CloudFinOpsKit Azure Tool and AWS Tool ship a Cost Anomaly Watch module with the three time-series AI detectors built in — token spikes, output bloat, and blended-rate shifts, alongside the general spend detectors — while the AI Workloads module flags under-utilized PTU and provisioned-throughput deployments. Deterministic math with published thresholds (every flag recomputable by hand), projected monthly impact per finding, and Teams/Slack webhook alerts for High severity. Same methodology on Azure and AWS — see the cross-cloud methodology for the full spec.

Reading the detectors together

The four detectors triangulate. Tokens up + ratio flat + rate flat = a volume event; find the client. Tokens flat + ratio up = a verbosity regression; find the prompt or cap change. Tokens flat + rate up = a mix shift; find the routing change. Everything flat + PTU idle = a commitment sized for a launch that didn't happen. Each signature names its own fix — which is exactly what a dollar-only alert can't do.

FAQ

Won't the native anomaly services catch AI spend problems?

Only the crudest ones. Azure Cost Management and AWS Cost Anomaly Detection watch dollars per service, so a big token spike eventually flags as "Azure OpenAI is up" — a day late and with no indication whether it's volume, mix, or shape. Ratio bloat and mix shifts at flat spend levels typically never flag at all.

What's a healthy output:input token ratio?

There's no universal number — a summarizer might run 1:10, a generator 3:1. What matters is stability: your workload's ratio should be roughly constant week to week, which is why the detector watches for relative drift (≥ 30% rise) rather than an absolute value.

How fast can these detectors catch a problem?

Token metrics are near-real-time in Azure Monitor and CloudWatch, so a daily scheduled run catches volume and shape anomalies within a day — typically before the spend even appears in billing data. Rate-shift detection needs billing data and trails by the billing lag, still days ahead of the invoice.