Laptop displaying an analytics dashboard with usage charts and figures

Reducing LLM Costs with Prompt Caching

July 16, 2026 7 min read

Prompt caching has a price on the way in. On Anthropic's API, writing a prefix to the cache costs more than sending it normally, so a prefix that is never read again makes your bill higher, not lower. Most guides give the ordering rule (static text first) and the discount. They skip the break-even point, the edits that silently empty the cache, and the traffic shapes where caching loses.

This guide covers how a cache hit works, what a hit saves and a miss costs, which changes invalidate the cache, and how to verify in staging that the cache is working.

What gets cached

LLM APIs are stateless: every request carries the full system prompt, tool definitions, and history. A transformer builds key and value tensors for every input token, and prompt caching keeps those tensors for a short time. OpenAI's prompt caching guide says the cache "stores key-value (KV) tensors, not the tokens themselves" and covers the reusable prefix of the rendered context, including instructions, tool definitions, and conversation history.

A hit needs an exact match from the first token. The first difference ends the match, and every later token is processed at full price even if it is identical to a previous request. Caching does not change the output. Identical requests are still not guaranteed to produce identical text.

A cache write costs more than a normal request

Anthropic's prompt caching documentation prices cache writes at 1.25 times the base input price for the 5-minute lifetime and 2 times for the 1-hour lifetime. Reads cost 0.1 times the base price on most models. Run the numbers for one prefix, in units of its normal cost:

5-minute cache. Two requests inside the window cost 1.25 + 0.1 = 1.35, against 2.0 uncached. Caching wins from the second request.
1-hour cache. Two requests cost 2.0 + 0.1 = 2.1, against 2.0 uncached. Caching loses on the second request and wins from the third (2.2 against 3.0).
A prefix used once. You pay the premium and never collect the discount.

The practical rule: cache a prefix only if you expect it to be read at least twice inside its lifetime, and choose the 1-hour option only if you expect three or more reads. A tool-using agent that makes several model calls in one turn clears that bar by itself. A batch of unrelated one-off requests does not.

The lifetime is not fixed from the first write. By default the cache lasts 5 minutes and, per the same documentation, "is refreshed for no additional cost each time the cached content is used." Steady traffic keeps it warm for free. Traffic with gaps longer than the lifetime pays the write premium again each time.

What empties the cache

Order matters because Anthropic builds the prompt as tools, then system, then messages. A change at one level invalidates that level and every level after it. The documentation lists the triggers:

Any change to tool definitions (names, descriptions, parameters) invalidates tools, system, and messages.
Changes to the system prompt invalidate system and messages.
Changing tool_choice, adding or removing an image anywhere in the prompt, or changing thinking parameters invalidates the messages cache.

The costly mistake is a per-user or per-request tool list. If your agent exposes different tools depending on the caller, or reorders the same tools between requests, the first thing in the prompt differs and nothing after it can hit. Sort tools deterministically, keep one tool list per tenant tier, and keep per-request data (date, user name, request ID) out of the system prompt and in the final user message.

Separately, the documentation notes that a cache entry becomes available only after the first response starts. If you fan out ten parallel requests that share a prefix, all ten miss and all ten pay the write premium. Send one request, wait for its first response, then send the rest.

Provider setup

Anthropic. You mark breakpoints with cache_control. Each model has a minimum cacheable length, from 512 to 4,096 tokens depending on the model, and shorter prompts are silently not cached with no error. Check the table for your model. The system looks back at most 20 positions from each breakpoint, so a long agent transcript needs a breakpoint near the end, not only one at the top.

OpenAI. Caching is automatic for prompts above the minimum length, and there is no marker to add. Do not invent one. The prompt_cache_key parameter helps related requests land on the same cache, and OpenAI advises staying near 15 requests per minute across all prefixes that share a key, since traffic above that can overflow to other machines.

Providers also differ in what they report. OpenAI reports cached tokens as a subset of the input count, while Anthropic's input_tokens covers only the tokens after the last cache breakpoint, so its total input is cache reads plus cache writes plus input_tokens. If you compare cost across providers, normalize those fields first.

The 20-block lookback that long agent turns outrun

The 20-position lookback has a failure case that the Anthropic caching documentation works through, and it hits tool-heavy agents hardest. For each breakpoint the system checks at most 20 positions backwards for an entry that an earlier request already wrote, and stops if it finds none. Writes happen only at breakpoints, so the lookback cannot find stable content that sits behind a breakpoint nobody wrote at. The docs' own example is a conversation that reaches 35 blocks with a breakpoint on block 35: the check covers blocks 35 down to 16, misses the entry written at block 15, and the request pays for a fresh write with no read. The documented fix is a second breakpoint placed closer to the earlier write from the start, so that a write accumulates there before you need it. On the Claude API a run of consecutive tool_use blocks counts as one position, and so does a run of consecutive tool_result blocks, which helps parallel tool calls but not a turn of many sequential ones.

Two corollaries follow from the same page. Put the breakpoint at the end of the static prefix, never on a block that changes per request: with a timestamp in the middle of the prompt, the docs show the prefix hash differing on every request, so you pay a write each time and never get a read. And log cache-write tokens in their own column, because they bill at a premium and folding them into ordinary input tokens makes the cost report wrong in the direction that hides a broken cache.

When caching does not replace retrieval

A common suggestion is to load the whole knowledge base into a cached prefix and drop the vector database. It works for a small, stable, single-audience document set, such as one operating manual. It fails in three cases.

Permissions. A shared prefix means every user's request carries every document, including ones they may not be allowed to read. Retrieval can filter by the caller's access. A shared cached prefix cannot.
Churn. Any edit to any document changes the prefix and forces a fresh write of the whole thing at the write premium.
Size. A cached read is cheaper but still scales with the prefix length, and long contexts add latency.

Use caching for what is genuinely shared and stable: instructions, tool schemas, style guides, few-shot examples. Use retrieval for content that varies by question or by user.

Privacy and isolation

Anthropic states that different organizations never share caches, even with identical prompts, and that the Claude API isolates caches by workspace; Bedrock isolates at the organization level. Within your own account, separate caches by the groups that need separate system prompts. HR and Engineering assistants with different instructions get different prefixes, so group requests by prefix to keep each one warm.

Cached state is still held by the provider. OpenAI's guide notes that caching may store encrypted KV tensors in GPU-local storage, and that organizations with Zero Data Retention default to in-memory retention where a model supports both modes. Check the retention mode for your account before putting personal data in the cached prefix, and keep personal data in the final message where you can.

Common pitfalls

Timestamps and IDs at the top. A date in the first line of the system prompt guarantees a miss. Put it at the end.
Gaps longer than the lifetime. Low-traffic tools expire the cache between requests and pay the write premium repeatedly. If the gaps are irregular, caching may cost you money. A keep-alive request every few minutes costs a read, but only pays if the real requests it protects would otherwise each pay a write.
Breakpoints after every sentence. Place them after large, stable blocks, and stay inside the provider's limit on the number of breakpoints.
Silent skips. A prompt shorter than the minimum is not cached and returns no error. Teams discover this only on the invoice.

Verify it in staging

Do not trust the dashboard average. Send a prompt that clears the minimum length twice, spaced inside the lifetime, and read the usage block of each response. On Anthropic the first response should show tokens in cache_creation_input_tokens and the second in cache_read_input_tokens. On OpenAI look at the cached token count in the prompt token details. If the second response shows zero cached tokens, diff the two rendered prompts byte by byte, looking for a timestamp, a reordered tool, a changed image, or a request ID.

Then run your real traffic shape for an hour and compute the hit rate per prefix. Report writes and reads separately. A prefix with many writes and few reads is the one costing you extra.

Next steps

List your largest static blocks and order the prompt as tools, system, shared references, then per-request data. Estimate reads per write for each prefix using the break-even figures above. Turn caching on only for prefixes that clear it, and review writes against reads after a week of real traffic.

References

OpenAI prompt caching guide: KV-tensor storage, prompt_cache_key, retention modes.
Anthropic prompt caching documentation: write and read multipliers, minimum lengths, invalidation rules, concurrency, and isolation.

About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation with the cost and latency controls described above, at published fixed prices. Schedule a call to discuss your next project.