Why I put a gateway in front of every LLM call
The first service that calls a model provider does it directly, and that is fine. By the fourth, each one holds its own provider key, has its own idea of a retry policy, and nobody can say what the company spends on models until the invoice arrives.
A gateway is one service that sits in the path of every model call. It is not clever. It holds the keys, counts the money, enforces limits, writes down what happened and moves traffic when a provider is down. Most of that is ordinary plumbing. One part, retrying without paying twice, is subtle enough that it deserves most of this post.
The shape
The gateway speaks an API the clients already know. The OpenAI-style chat completions shape is the common choice, because most SDKs and frameworks can talk to it, so moving a service behind the gateway is a change of base URL and key, not a rewrite. Services authenticate to the gateway with a key of their own. Only the gateway authenticates to providers.
Provider keys in one place
When every service holds a provider key, rotating it means touching every service, and a leak from any of them is a leak of the account.
With a gateway, provider keys sit in one secret store and are read by one service. Each team gets a gateway key that is worth nothing outside the gateway and can be revoked on its own. That key also answers which team made a request, and everything else in this post depends on that answer.
Budgets that actually stop spending
A monthly invoice is a report, not a control. The failure that hurts is not an outage; it is a loop in a batch job that keeps calling the model all weekend. A per-team budget with a hard cap turns that into a rejected request on Saturday morning instead of a surprise at month end.
The obvious implementation is wrong: read the spend, compare with the cap, call, add the cost. Two requests that arrive together read the same balance, both pass, and both spend. With agents and batch jobs, that is the normal case.
What works is reserving before the call and reconciling after it. Before calling the provider, estimate the
worst case: prompt tokens plus max_tokens, at the model's price. Reserve that amount in one conditional
statement:
UPDATE team_budgets
SET reserved_micros = reserved_micros + @estimate
WHERE team_id = @team
AND spent_micros + reserved_micros + @estimate <= cap_micros
RETURNING cap_micros - spent_micros - reserved_micros AS remaining_micros;
No row back means the team is out of budget, and the request is refused before any money is spent. When the response arrives, release the reservation and add the real cost from the provider's usage numbers. The database does the serialisation; there is no read-then-write window.
Keep money as an integer count of small units, millionths of a currency unit here, never as a floating-point number. A cap that is "about" right at the boundary is not a cap.
Rate limits, yours and theirs
There are two kinds. Providers limit your account, and every team shares that limit, so one team's backfill can starve another team's user-facing feature. And you want your own limits per team, so that the sharing is a decision rather than an accident.
A token bucket per gateway key covers most cases. Limit tokens, not only requests: providers count tokens, and one large document can cost as much as a hundred small questions.
Logging, and the prompt problem
For every request the gateway records who sent it, which model and provider served it, input and output tokens, cost, latency, status and a request id it also returns to the caller. With that in a table, spend by team and model is a query instead of a project, and the same data as OpenTelemetry traces connects a slow user request to the model call that made it slow.
Prompts are the hard part. Logging them makes a bad answer much easier to debug. But a prompt contains whatever users typed and whatever documents were retrieved: personal data, customer files, sometimes credentials pasted by mistake. Logging prompts creates a second copy of all of it, usually in a system with wider access and longer retention than the original.
My defaults: metadata always, prompt text off. Where a team needs prompts for debugging, store them redacted, with short retention and access limited to that team. It is a trade-off and the gateway is the right place to make it, once, instead of in every service.
Failover between providers
Callers ask for an alias such as fast or default, and the gateway maps it to an ordered list of backends.
When the first is down or rate limited, the next one serves the request, and a circuit breaker stops sending
traffic to a provider that keeps failing.
Be honest about what failover is: a different model answers differently, and may support different structured output features. The backends behind one alias should be ones you have tested with your own prompts, not anything that accepts the same request body. Structured extraction is the most fragile case; see Getting structured data out of an LLM.
The subtle part: never paying twice
A completion that a provider generated has been billed, whether or not your gateway received the response. A retry or a failover after that point pays for the same answer again. So the question before every retry is not "did it fail?" but "do I know it didn't run?".
First, only one layer gets to retry. The client SDK, an HTTP resilience handler and the gateway's own failover can each retry, and stacked they multiply: three attempts at each of two layers is up to nine calls for one request. The gateway owns retries; retries in the transport underneath it are switched off, and clients are asked to leave theirs off too.
Second, fail over only on errors that prove the first request did not run:
using System.Net;
// True only when the failure shows the provider never generated a completion,
// so trying the next provider cannot produce a second billable one.
static bool SafeToFailOver(HttpResponseMessage? response, Exception? error) =>
(response, error) switch
{
({ StatusCode: HttpStatusCode.TooManyRequests }, _) => true,
({ StatusCode: HttpStatusCode.Unauthorized or HttpStatusCode.Forbidden }, _) => true,
(null, HttpRequestException
{
HttpRequestError: HttpRequestError.NameResolutionError
or HttpRequestError.ConnectionError
or HttpRequestError.SecureConnectionError
}) => true,
_ => false, // 5xx, timeouts, cut-off streams: it may have run
};
A 429 is the provider saying it refused the request, and a 401 or 403 says it was rejected before any work
started, usually because that provider's key is wrong. A failed DNS lookup, connection or TLS handshake means
the request never arrived. Those are proofs. A 400 is also proof, but failing over doesn't help: the request
itself is wrong, and the next provider will reject it as well. A 500 says something broke, not when it broke.
Check what your provider documents for its own status codes before you move anything out of the false arm.
Timeouts are the ambiguous case that matters most. When the gateway gives up waiting, the provider may still be generating, and it will bill for what it generates. Failing over at that point means paying twice and hoping that is rare. My default is to return an error for ambiguous failures and charge the reservation as spent, because I can't prove it wasn't. If latency matters more than cost for some alias, fail over on timeouts there deliberately, and record both attempts in the ledger so the bill and your numbers agree.
Streaming narrows the window further: once the first chunk has reached the caller, switching providers means half an answer from each. Failover happens before the first byte or not at all.
Third, make the client's own retries safe. A client that times out waiting for the gateway will send the request again. With an idempotency key on each logical request, the gateway stores the outcome under that key: a repeat while the first is running waits for it, and a repeat after it finished gets the stored response, without running anything. Don't assume the provider will deduplicate for you.
What I built
I built one, Tollgate. Its README covers the parts above: an OpenAI-compatible endpoint, budgets reserved in one conditional statement and reconciled after, a failover walk that owns retries with transport retries disabled, and prompts redacted before storage. Its failover test doesn't trust the gateway's own count either: it asks the test's mock providers how many completions they actually generated.
When not to bother
A gateway is another hop on the critical path of every AI feature you have. If it is down, all of them are down, so it needs more than one instance. With one service and one team, a key in a secret store and a budget alert from the provider may be enough.
The moment there is a second team, a second provider, or a question about spend nobody can answer, I would rather have the gateway. Keep it boring: it should forward, count and refuse, not rewrite prompts or hold business logic.
