Caching LLM calls, and when a semantic cache lies
LLM calls are slow, billed per token, and a lot of them repeat. The same support question arrives a hundred times a day, the same document gets summarised on every page view. Caching is the obvious fix, and there are two kinds. One is boring and safe. The other saves more and can answer a question nobody asked.
This post is about both: what has to go into an exact-match cache key, why a semantic cache can return the right answer to the wrong question, and how to scope, expire and measure it so you find out before your users do.
Exact match: hash everything that changes the answer
An exact-match cache returns a stored response only when the request is identical. The whole job is deciding what "identical" means, and the rule is simple: anything that can change the output goes into the key. Forget one thing and you serve an answer that was computed under different conditions.
That list is longer than the prompt:
- The model, down to the deployment or version. A new model version gives different answers to the same prompt.
- Sampling parameters: temperature, max tokens, anything else you send.
- The system prompt, by version. When you change the instructions, old answers are stale.
- The scope: tenant, and the user too if the answer depends on who asks.
- Every message in the conversation, not just the last one.
- Retrieved chunks and tool results, by ID and version. In RAG the answer depends on what was retrieved, so the same question over a changed document is a different request.
Temperature deserves a note. Above zero, the model would give a different answer each time, and a cache freezes one sample. For a FAQ that is fine, even desirable. For "give me five fresh ideas" it defeats the point. That is a product decision, not a caching detail.
I put all of it in one record and hash its serialized form:
public sealed record Turn(string Role, string Text);
public sealed record LlmRequest(
string Model,
float Temperature,
string PromptVersion, // version of the system prompt, not its text
string TenantId,
IReadOnlyList<Turn> Messages,
IReadOnlyList<string> Sources); // retrieved chunk or tool result ids, with versions
static string CacheKey(LlmRequest r)
{
byte[] hash = SHA256.HashData(JsonSerializer.SerializeToUtf8Bytes(r));
return $"llm:{r.TenantId}:{Convert.ToHexString(hash)}";
}
Hashing also follows Microsoft's guidance for HybridCache keys: don't put raw user input in a key, and
stay under the 1,024-character default limit. A prompt is both user input and long. A hash is neither.
Storing it with HybridCache
HybridCache from Microsoft.Extensions.Caching.Hybrid fits this well. It keeps an in-memory cache, uses
a registered IDistributedCache such as Redis as a second level, and lets only one concurrent caller per
key run the factory. Fifty identical prompts arriving at once become one model call on that instance.
public async Task<string> AskAsync(LlmRequest r, CancellationToken ct) =>
await cache.GetOrCreateAsync(
CacheKey(r),
async token =>
{
List<ChatMessage> messages =
[.. r.Messages.Select(m => new ChatMessage(new ChatRole(m.Role), m.Text))];
var options = new ChatOptions { ModelId = r.Model, Temperature = r.Temperature };
ChatResponse response = await chat.GetResponseAsync(messages, options, token);
return response.Text;
},
new HybridCacheEntryOptions { Expiration = TimeSpan.FromHours(6) },
tags: [$"tenant:{r.TenantId}", .. r.Sources.Select(s => $"source:{s.Split('@')[0]}")],
cancellationToken: ct);
public ValueTask OnSourceChangedAsync(string sourceId, CancellationToken ct) =>
cache.RemoveByTagAsync($"source:{sourceId}", ct);
Each entry is tagged with its tenant and the documents it was built from. When a document changes,
RemoveByTagAsync invalidates every answer that used it. Two details from the docs: tag removal is
logical, the old values stay stored but are treated as misses, and it doesn't reach the in-memory cache
of other servers. Keep LocalCacheExpiration short if several instances serve traffic.
If you only need one process, my own OrionCache does the same single-flight
GetOrCreateAsync with tag invalidation and TTLs that run on a fake clock in tests. Version 1.0 is
in-memory only, so its tags are local to one instance too.
One thing I would never cache, by either method: a response that asked for a tool call with side effects. Cache the read, never the act.
Semantic caching: nearest is not the same
An exact-match cache misses on "how do I reset my password" after storing "how can I reset my password". A semantic cache embeds each prompt, finds the nearest stored one, and returns its answer if the two are similar enough. Paraphrases hit, so the hit rate goes up.
The problem is what "similar" measures. Embeddings put texts about the same topic close together. They don't promise that two texts mean the same thing, and the cases where they don't are exactly the ones that hurt:
- Negation. "Cancel my order" and "don't cancel my order" differ by one word and are about the same thing. They can land very close.
- Entities. "Refund policy for Germany" and "refund policy for France" are one token apart.
- Scope. "What is my balance?" is the same text for every user, with a different correct answer for each.
- Time. "What changed this week?" means something new every Monday.
An exact-match cache serves a wrong answer only when the key is missing something. A semantic cache serves a different question's answer by design, whenever the threshold lets it. So the work is limiting where that can happen.
Scope first, then similarity
Only the user's free-text question should be fuzzy. Everything else from the key list above stays exact, and the similarity search runs only inside that scope:
If the cache lives in Postgres, it is a pgvector query with the scope in the WHERE clause:
SELECT response, prompt, embedding <=> $1 AS distance
FROM llm_semantic_cache
WHERE tenant_id = $2 AND model = $3 AND prompt_version = $4
AND expires_at > now()
ORDER BY embedding <=> $1
LIMIT 1;
I compare the distance to the threshold in C#, not in SQL. The pgvector README advises against putting a distance condition next to the index scan (it suggests a materialized CTE instead), and the C# check is simpler and lets me log the near misses. And with an approximate index, a filtered query can come back empty even when a match exists, for the reasons in the pgvector post. For a cache that is safe, it just becomes a miss.
Some requests should never reach the semantic path at all: answers built from personal data, time-dependent questions, anything that leads to a tool call, and follow-ups in a conversation. "And the second one?" means nothing without the history, and two identical follow-ups in different conversations are different questions.
Choosing the threshold
There is no universal number. Distances depend on the embedding model and on your domain, so a threshold copied from somewhere else is a guess. I would pick it from data:
- Collect pairs of real prompts and label each pair: same answer, or different answer.
- Include hard negatives on purpose: negations, swapped entities, different dates.
- Compute the distance for every pair and look at where the two groups overlap.
- Choose the threshold by the false-hit rate you can accept on the hard negatives. The hit rate is whatever is left.
Changing the embedding model changes every distance, so it means a new threshold and an empty semantic cache. Vectors from two models can't be compared anyway.
TTLs and invalidation
Every entry needs a TTL, set by how stale an answer is allowed to be, not by how much you want to save.
On top of that, invalidate on change. The exact cache uses tags. The semantic table stores the source IDs
each answer was built from and deletes by them: DELETE FROM llm_semantic_cache WHERE $1 = ANY(source_ids).
A new prompt version needs no cleanup at all. It is part of the scope, so old entries simply stop matching
and expire.
Measuring the false-hit rate
Hit rate alone is a vanity metric. A threshold that accepts everything has a perfect one. What matters is how often a hit was wrong, and you can only know that by checking:
- Log every semantic hit with both prompts and the distance. Read a sample regularly. Wrong hits are usually obvious to a person.
- Shadow a small share of hits. Call the model anyway, compare its answer with the cached one, and flag the pairs that disagree for review.
- Log near misses too, the ones just above the threshold. They tell you whether loosening it would be safe.
Track the false-hit rate next to the hit rate, per tenant and per question type. My default is exact-match caching everywhere it applies, and a semantic cache only for narrow, stateless questions where I have that number and can live with it.
