The bill arrives on the second of the month. You open it, then you open the MRR dashboard, and the two numbers sit closer together than they did in June. Nothing broke. No runaway loop shipped. Usage grew the way you wanted it to, and model spend grew alongside it, linearly, because that is what per-token pricing does.
Most writing about AI cost optimization stops at "set a usage limit." A limit is a circuit breaker. It tells you the house is burning without making the house cheaper to heat. Teams whose gross margin survives contact with an LLM feature run four layers underneath that limit: a prompt cache, a semantic cache trusted for one narrow slice of traffic, a router that sends most requests to a cheap model, and a fallback chain that degrades instead of failing.
I have shipped these layers into production SaaS. Three of them I would build the same way again. One I would scope far more tightly than I did the first time.
Why does AI cost optimization fail at the usage-limit stage?
Usage limits cap your worst month without lowering your average one, because the cost lives in the shape of every request rather than in the count.
Picture a typical request in a B2B product. A system prompt. A tool schema. Tenant configuration. Two or three thousand tokens of retrieved context. Then a user message of forty tokens. You pay full input price for all of it, and the first four pieces are byte-identical to the last ten thousand requests you served. Every request. Forever.
The redundancy inside each unit of volume is the leak, and volume is only the multiplier on top of it. A usage limit does nothing about redundancy, which is why the first month after you set one looks almost exactly like the month before, minus the single Tuesday where a customer ran a bulk import.
How many of your requests are actually novel? In a product with a fixed system prompt and a fixed schema, almost none of them, measured in tokens. The novel part is the last forty tokens and whatever comes back.
Prompt caching pays before you change a line of product code
Prompt caching bills a repeated prefix at roughly a tenth of the normal input rate, so the second request that reuses it already costs less than not caching at all.
Anthropic's documentation puts the arithmetic on the table. A five-minute cache write bills at 1.25x the base input rate, a one-hour write at 2.0x, and every read against a live cache entry at 0.1x (Claude platform docs). One write, then reads at a tenth of list. OpenAI's cached input reads land in the same neighbourhood, at 10% of the standard input rate across the GPT-5.6 tiers (CloudZero).
There is a second effect that nobody puts on the pricing page. Cache read tokens do not count against Anthropic's input-token-per-minute rate limits, so a well-placed breakpoint buys throughput headroom on top of the discount. If you have ever watched a batch job crawl because of ITPM rather than compute, that line is worth more than the money.
So cache everything stable and you are done. Actually, that skips the part that bites. Cache invalidation runs in strict order, tools then system then messages, and a change at any level invalidates that level and everything after it. Version a tool definition and you have thrown away the system-prompt cache for every tenant at once. The placement rule from the same docs is the one to internalise: put the breakpoint on the last block whose prefix is identical across requests, never on a block carrying a timestamp or per-request context. You get four explicit breakpoints, and automatic caching consumes one of them.
Here is how the four layers compare once they are all in place.
| Layer | What it costs when it works | Realistic hit rate | Failure mode |
|---|---|---|---|
| Prompt cache (prefix) | 10% of base input price | High on fixed prompts and schemas | Silent miss: you paid the write premium and got nothing back |
| Semantic cache (embedding) | One vector lookup, near zero | 23% to 88%, entirely threshold-dependent | A wrong answer returned with HTTP 200 |
| Model routing | Cheap-tier token price | Applies to every request | An under-powered model answers a hard question confidently |
| Fallback chain | The second provider's list price | Only on failures | A retry storm that deepens the outage it was meant to survive |
Semantic caching is the layer that quietly returns wrong answers
This is the layer I would scope far more tightly than I did the first time, and the numbers explain why better than any war story.
Portkey published production threshold data in April 2026 that belongs taped to the monitor of anyone about to ship one. At a cosine similarity threshold of 0.80, they measured an 87.6% hit rate with accuracy falling to 91.8%. Raise the threshold to 0.99 to protect that accuracy and the hit rate collapses to 23.5%, dragging total cost savings down to 15.8% (Portkey). At the loose end roughly one answer in twelve was wrong. At the safe end you were barely saving anything.
Now the failure mode. A bad semantic cache hit does not throw. It returns a plausible answer with a 200 and a latency number your dashboard will happily celebrate. There is no error rate to alert on, because from the system's point of view nothing failed. Compare that to a cache miss, which just costs you a normal API call. The asymmetry is brutal, and it is why the two layers deserve very different levels of trust.
Portkey's own guidance is to start around 0.95, backtest against roughly 5,000 real queries, and treat the threshold as a per-workload knob instead of a constant. That posture is right. It also means a semantic cache is not a weekend feature; it is infrastructure with an evaluation suite attached, and it needs re-validating whenever your embedding model changes.
Where it does earn its place is a bounded question space. Support macros and glossary lookups. The forty onboarding questions every new tenant asks in their first week, which in B2B SaaS is a surprisingly stable set. Cache those forty and route everything else straight past the cache.
Which model should get the request?
Route on a cheap measurable signal, then send only the residue you cannot confidently classify to the frontier model.
The academic version is RouteLLM, from Ong and colleagues at UC Berkeley and LMSYS, which trains a router on preference data and reports cost reductions of over 2x with no compromise in response quality on standard benchmarks (arXiv 2406.18665). The production version is usually simpler and nearly as effective, because in a real product you know things a benchmark cannot: which feature made the call, which tenant plan the caller is on, whether a tool result is already in context.
A routing rollout that survives contact with users looks like this:
- Ship the cheap tier as the default, not as an experiment. Every request goes to the small model unless a rule sends it elsewhere. Defaults decide outcomes, and a router that defaults to the expensive model will quietly stay there, because nobody gets paged for spending money.
- Escalate on evidence, not on vibes. A structured-output parse failure, a refusal, a confidence field below threshold, a user hitting regenerate. All of those are signals you can log today.
- Measure the escalation rate per feature. A feature escalating 60% of the time does not need a better router; it needs to be pinned to the strong model and priced accordingly.
- Re-run the arithmetic every quarter. Prices move underneath you.
That last point is not theoretical. On 30 July 2026 OpenAI cut GPT-5.6 Luna by 80%, from $1/$6 to $0.20/$1.20 per million input and output tokens, and cut Terra 20% in the same move, while the Sol flagship held at $5/$30 (CloudZero). A cheap tier getting five times cheaper overnight changes which requests are worth escalating. Any threshold tuned in June was tuned against arithmetic that has since stopped being true. Put a calendar reminder on your routing config the way you would on a TLS certificate. The same discipline I apply to an AI-augmented development workflow applies here: the tooling changes faster than your assumptions about it.
Fallbacks are a cost decision, not only an uptime one
Retrying a 429 immediately runs a load test against a provider that already asked you to stop, and the failed calls still burn quota. Exponential backoff with jitter, then a second provider, then a degraded response that admits it is degraded. The saving hides in that third step. A graceful "try again shortly" costs nothing, while a retry storm bills you three times for one answer nobody received.
What stops one tenant from eating your whole AI budget?
Per-tenant ceilings enforced server-side, checked before the model call rather than after it, with an hourly limit and a daily one.
On Callidus OS, the multi-tenant clinic SaaS I built solo on React, TypeScript and Firebase, the assistant sits behind exactly that. Calia runs on Vertex AI through the @google/genai SDK with 30-per-hour and 100-per-hour rate limits enforced inside the Cloud Function, thirteen injection regex patterns in front of the prompt, and a 90-day cleanup job for GDPR. Fourteen tenants were onboarded by the time the engagement closed in June 2026. Everything lives under a per-tenant Firestore path with tenantId and role carried in JWT claims, which means the budget check has a tenant to attribute spend to before it does anything else.
The ceiling was never about a clinic abusing the product. It was about one clinic's enthusiastic receptionist discovering the assistant on a slow Tuesday and running ninety queries in an afternoon, on a £15/month Starter plan. Enforce it client-side and you have written a suggestion. The number that matters is not your total AI bill; it is your AI cost per tenant per month, sorted descending, because the tail is where the margin goes. I wrote more about which AI features actually earn their place in business SaaS and the same rule holds: an unmetered feature is an unpriced feature.
Build it in this order
- Instrument. Log input tokens, output tokens, cached tokens, model, tenant and feature on every call. Without this, everything below is guesswork with a confident tone.
- Turn on prompt caching. Highest saving per hour of work, no product change, no quality risk. Verify it landed by checking that cache read tokens are non-zero rather than trusting the config.
- Add per-tenant ceilings. Cheap to build, and it converts your worst-case bill from unbounded to arithmetic.
- Add routing, cheap tier first. Then watch escalation rates per feature for two weeks before touching a threshold.
- Add fallbacks with backoff and jitter. Uptime insurance that also stops retry storms from billing you.
- Consider a semantic cache last, for a bounded question set only. With an eval suite. With a threshold you backtested. Or skip it, and accept that you have left maybe 10% on the table in exchange for never shipping a confidently wrong answer.
If you are still picking the substrate underneath all of this, the SaaS MVP stack guide covers the layer below: what to build the product on before there is an AI feature to meter.
Open your most recent provider invoice and answer one question. What percentage of your input tokens were cache reads last month? If you cannot answer it from a dashboard in under a minute, that is your whole project for this week, and every other layer here stays guesswork until you can.
