Caching LLM calls, and when a semantic cache lies
· 8 min read
LLM calls are slow, billed per token, and a lot of them repeat. The same support question arrives a hundred times a day, the same document gets summarised on every page view. Caching is the obvious fix, and there are two kinds. One is boring and safe. The other saves more and can answer a question nobody asked.
This post is about both: what has to go into an exact-match cache key, why a semantic cache can return the right answer to the wrong question, and how to scope, expire and measure it so you find out before your users do.
