The economics of semantic caching for LLM traffic

Every LLM request you send to a provider costs tokens. Many of those requests ask essentially the same question — “summarize this policy doc,” “what’s our refund policy,” “translate this paragraph.” Exact deduplication catches a small fraction of them. Semantic caching catches the rest.

Why exact caching is not enough

Exact deduplication compares the literal prompt string. If two users ask “What is our PTO policy?” and “what is our PTO policy?” — different capitalization — they generate separate provider calls. If one user asks “What’s our vacation policy?” and another asks “How many vacation days do I get?”, both questions may have identical answers but an exact cache will never connect them.

Semantic caching embeds the prompt and compares the resulting vector against a cache index. Prompts that mean the same thing — regardless of phrasing, spelling variation, or word order — resolve to nearby vectors and hit the cache. This is the difference between a marginal cache hit rate and a meaningful one.

What semantic caching actually does

When a request arrives at ManyLayers Gateway, the gateway embeds the prompt using a configured embedding model and compares the resulting vector against a cache index. If the cosine similarity between the incoming prompt and a cached prompt exceeds a configurable threshold, the gateway returns the cached response without forwarding the request to a provider.

The threshold is the most important configuration decision. A threshold of 0.99 behaves like fuzzy deduplication — only near-identical prompts hit the cache. A threshold of 0.92 gives meaningful semantic coverage but risks serving responses that are accurate for a similar but not identical question. For most enterprise deployments, 0.95–0.97 is the practical starting point.

The cost math

Consider a team generating 50,000 gateway requests per day at an average cost of $0.008 per request. A 20% semantic cache hit rate saves $80 per day, or roughly $29,000 per year — before accounting for latency improvements from cache hits returning in under 10ms versus the 300–2,000ms of a live provider call.

At scale, the savings grow non-linearly because knowledge bases and internal tools tend to generate highly repetitive prompt distributions. A support bot answering questions about a 200-page policy manual will see much higher hit rates than a free-form coding assistant.

The embedding cost is not zero, but it’s small relative to generation cost. A text-embedding-3-small call costs roughly $0.00002 per prompt — 400x cheaper than a typical GPT-4o completion. For a workload with a 25% cache hit rate, you’d spend about 40x less on embedding costs than you save on generation costs.

Setting up semantic caching in ManyLayers Gateway

ManyLayers Gateway supports semantic caching at the alias level. You configure an embedding model (any of the supported providers, or a locally deployed model via the Deploy module), a similarity threshold, and a TTL for cached entries:

aliases:
  support-bot:
    semantic_cache:
      enabled: true
      embedding_model: openai/text-embedding-3-small
      threshold: 0.96
      ttl: 3600          # seconds
      max_entries: 50000

The embedding model runs inference on every uncached prompt. For cost-sensitive deployments, a small local embedding model deployed via the Deploy module can handle this with negligible latency overhead and no per-request charge.

For larger deployments, deploy a dedicated embedding service rather than routing embedding calls through the same alias as completions. This prevents embedding traffic from consuming budget headroom allocated for generation, and allows you to scale the embedding service independently.

What to measure

Cache efficiency is not just a hit rate. Track these metrics:

• Hit rate by alias — different workloads have different semantic density. Don’t average across all traffic.
• False positive rate — sample cache hits and evaluate whether the returned response was actually appropriate for the prompt. Log disagreements to a review queue.
• Latency distribution — P99 latency for cache hits should be well under 50ms. If it isn’t, your embedding store or index is undersized.
• TTL expiry patterns — if most entries expire before being hit, your TTL is too short or the prompt distribution is too diverse for caching to help.
• Cost savings by alias — the Analytics dashboard shows estimated savings in dollars per day per alias. Use this to prioritize which aliases to optimize first.

Interaction with guardrails and PII firewall

Semantic caching runs after the PII firewall strips sensitive fields. The cache index never stores raw PII — it stores the embedding of the sanitized prompt and the sanitized response. This means cached responses may have redaction markers. If your application requires the original values to be filled back in, handle that reconstruction downstream.

Guardrails run on cache hits as well as live responses by default. You can configure guardrail-skip on trusted cached entries if your policy allows it, but this is not recommended for compliance-sensitive workloads.

When caching doesn’t help

Semantic caching delivers the most value on narrow, repetitive workloads: support bots, document Q&A, policy lookup, and structured data extraction from templated inputs. It delivers little value on open-ended generation (creative writing, code synthesis with highly variable context) and real-time grounded queries where freshness matters more than cost.

Measure your workload’s semantic entropy before deploying caching — ManyLayers Gateway can log prompt embeddings without caching them, letting you estimate hit rates at various thresholds before committing to a configuration. Run in “dry run” mode for two weeks: the gateway computes similarity scores and logs what the hit rate would have been, but still forwards all requests to providers. This gives you a reliable projection before you commit to a threshold.