LLM Semantic Caching: Reusing Answers Without Going Stale

A semantic cache is the riskiest way to cut LLM cost, and for many workloads the cheaper options win before you reach it. This post ranks the options, then covers what you must build if a semantic cache is still the right call: scoped lookups, entity checks, a measured threshold, and dependency-based invalidation.

Try the cheaper caches first

An exact cache reuses a value when the key matches. Provider prompt caching reuses processing of a repeated prefix, and the model still answers the current request. A semantic cache stores a finished answer and returns it for a request that merely looks similar. Only the last one can return a confident wrong answer.

Provider prompt caching is often the better first step. Anthropic's prompt caching documentation prices cache reads at 0.1x the base input price for most models (lower for some newer ones), with a 5-minute write at 1.25x and a 1-hour write at 2x. A long system prompt or shared document that repeats across requests gets most of the input saving, and the answer is generated fresh for the caller, so there is no stale-answer or cross-user risk. The multipliers also stack with the Batch API discount.

The second option is exact caching of deterministic work. Cache deterministic work such as language detection or string classification under a SHA-256 hash of the input, with no embeddings. The key must contain everything the result depends on and nothing else. Language detection depends only on the text, so its key leaves everything else out. A translation also depends on the target language, so its key must include it. Deciding what each cached value truly depends on is the same discipline a semantic cache needs, minus the similarity guesswork. Provider-side prefix caching has a routing dependency of its own. OpenAI's guide says cached states live on individual machines, that traffic above about 15 requests per minute can lead to overflow routing, and that on models before GPT-5.6 a busy group of requests should be split across several prompt_cache_key values to stay near that rate per key (OpenAI prompt caching).

Only when requests are phrased differently but need the same answer, the answer is stable, and the call is expensive enough does a semantic cache pay off. Compute that before you build it. Every request, hit or miss, pays for an embedding and a vector lookup, and a wrong hit costs more than a model call. If your measured hit rate on safe pairs is low, the layer adds latency and risk for a small saving.

Similarity does not separate right from wrong

Teams tune a distance threshold on a sample and ship it. The vCache paper (arXiv 2502.03771) argues that existing systems use one static threshold for all requests, which gives no formal correctness guarantee and can produce unexpected error rates. Its authors instead learn a threshold per cached prompt, with a user-defined error bound. You do not need their algorithm to take the lesson: a single global number copied from a library's docs has no error rate attached. Build a labeled set of close-but-different pairs (changed order number, negation, different role, a date boundary) and report the precision of accepted hits, not just hit rate. Re-measure whenever the embedding model or prompt normalization changes.

Some pairs should never share an answer whatever the score. "Refund status for order 8412" and "for order 8142" embed almost identically. Extract the material fields (entity IDs, jurisdiction, effective date, locale) with route-specific parsing, and force a miss when they differ or when one request has a field the other lacks.

Scope the lookup before the search

A shared vector index is not a shared authorization domain. Filter candidates by a server-built scope before comparing similarity: organization, an authorization fingerprint (effective permissions, not the user name), route, locale, model, system prompt version, tool contract version and knowledge version. Microsoft's semantic caching policy has a vary-by setting for this, and RedisVL supports tags and filters for multi-user caches. The cache keeps model output beyond the original request, so it falls under the same disclosure concerns as any stored output (OWASP LLM02:2025): encrypt it, limit access, and support deletion. Do not cache routes whose answers depend on live state or that trigger actions. A cached response may explain; it must never authorize a tool call.

A TTL only bounds how long an error lives

A one-hour TTL can serve an obsolete policy answer for 59 minutes after the policy changes, and a short TTL erases the savings. Store the dependencies with each entry (policy or document IDs, knowledge version, prompt version, tool schema version, model and embedding versions) and either bump a version key or delete affected entries when one changes. A practitioner issue on event-driven invalidation proposes version tags and stale-hit metrics; it is one team's design, not a standard.

Fail toward the model

An error, timeout, malformed entry, missing scope field or uncertain signature is a miss. Give the lookup a short deadline, and keep the rate limits that protected the model before caching, since Microsoft's guidance places rate limiting after the cache lookup so backend traffic stays bounded when the cache is down.

def answer(request, caller):
    policy = route_policy(request.route)
    if not policy.semantic_cache_allowed:
        return run_normal_path(request, caller)
    scope = build_verified_scope(request, caller)
    signature = extract_required_signature(request)
    if scope.incomplete or signature.uncertain:
        return run_normal_path(request, caller)
    for c in semantic_lookup(scope, embed(request.text), deadline_ms=40):
        if c.expired or c.scope != scope:
            continue
        if hard_fields_conflict(signature, c.signature):
            continue
        if c.distance > policy.validated_distance:
            continue
        return c.response
    response = run_normal_path(request, caller)
    if response.cacheable and response.dependencies_known:
        store(scope, signature, response, response.dependencies)
    return response

Rollout and tests

Run the labeled pairs offline, then run the cache in shadow mode in production without serving hits, and compare each would-be hit with the normal path's answer. Promote one low-risk route at a time. The negative tests that matter: same wording from a different organization, revoked permission, changed entity ID, a prior policy version, negated or time-sensitive wording, a cache outage, and a corrupt entry. Report accepted hits next to entity-guard rejections, scope rejections, fallback rate and sampled false hits, because a high hit rate says nothing when nobody counts wrong hits. Start with one read-only route and 100 request pairs, at least half of them hard negatives.

References

• Anthropic prompt caching
• vCache: Verified Semantic Prompt Caching
• Microsoft: Enable semantic caching for LLM APIs
• RedisVL: Cache LLM responses
• GPTCache
• OWASP LLM02:2025 Sensitive Information Disclosure
• Memzent.AI issue 11
• OpenAI: Prompt caching: cached states live on individual machines, with overflow routing above about 15 requests per minute.

---