Why I put a gateway in front of every LLM call
· 8 min read
The first service that calls a model provider does it directly, and that is fine. By the fourth, each one holds its own provider key, has its own idea of a retry policy, and nobody can say what the company spends on models until the invoice arrives.
A gateway is one service that sits in the path of every model call. It is not clever. It holds the keys, counts the money, enforces limits, writes down what happened and moves traffic when a provider is down. Most of that is ordinary plumbing. One part, retrying without paying twice, is subtle enough that it deserves most of this post.
