Streaming LLM answers to the browser with Server-Sent Events
A model writes its answer one token at a time. If your API waits for the last token before it returns anything, the user stares at a spinner for the whole generation, even though the first sentence was ready long before.
Streaming fixes that, and in ASP.NET Core it is now a few lines of code. The interesting parts are around those lines: which transport to use, what happens when the user closes the tab, what proxies do to a long response, and what "resume the stream" actually asks of your server.
Why stream at all
Streaming does not make the model faster. The total time to generate an answer is the same either way. What changes is the time to first token: the moment the user sees something happen. A reply that starts appearing immediately and takes a while to finish feels responsive; the same reply delivered in one piece at the end feels slow, and on a long answer it feels broken.
A user who sees the answer going the wrong way can also stop it early, which saves tokens if the server notices.
Why SSE and not WebSockets
An LLM answer flows in one direction: the server talks, the browser listens. Server-Sent Events is built
for exactly that shape. It is a normal HTTP response with Content-Type: text/event-stream that the
server keeps open and writes small text events into.
That gives you a few things for free:
- It is plain HTTP. Authentication, CORS, logging, rate limiting and your existing middleware all apply unchanged. There is no protocol upgrade for a proxy or a corporate firewall to refuse.
- The browser has a client built in.
EventSourceparses the format and dispatches events by name. - Reconnect is automatic. If the connection drops,
EventSourceopens it again, and sends the id of the last event it received in aLast-Event-IDheader, so the server can continue from there.
WebSockets earn their extra machinery when both sides send messages continuously, as in collaborative editing or games. A chat UI sends one question and receives one long answer; the question can be an ordinary POST.
One catch, and it shapes the design. EventSource can only make GET requests, with no body and no custom
headers. If you want to POST the question and read the answer from the same response, you read it with
fetch and parse the stream yourself, and you give up the automatic reconnect. I come back to that in the
section on resuming.
The endpoint
.NET 10 has SSE built in. System.Net.ServerSentEvents defines SseItem<T>, an event with a payload, an
event type, an optional id and an optional reconnection interval, and ASP.NET Core 10 adds
TypedResults.ServerSentEvents, which takes an IAsyncEnumerable of them and writes the response. It works
in minimal APIs and controllers. On the other side, IChatClient.GetStreamingResponseAsync from
Microsoft.Extensions.AI returns an IAsyncEnumerable<ChatResponseUpdate>. The two fit together directly:
using System.Net.ServerSentEvents;
using System.Runtime.CompilerServices;
using Microsoft.Extensions.AI;
app.MapPost("/chat", (ChatRequest request, IChatClient chat, HttpResponse response,
CancellationToken ct) =>
{
response.Headers["X-Accel-Buffering"] = "no"; // ask nginx not to buffer
return TypedResults.ServerSentEvents(StreamAnswer(chat, request.Question, ct));
});
static async IAsyncEnumerable<SseItem<string>> StreamAnswer(
IChatClient chat, string question, [EnumeratorCancellation] CancellationToken ct)
{
int seq = 0;
await foreach (ChatResponseUpdate update in
chat.GetStreamingResponseAsync(question, cancellationToken: ct))
{
if (string.IsNullOrEmpty(update.Text))
continue; // usage, tool calls and other non-text updates
yield return new SseItem<string>(update.Text, "delta") { EventId = (seq++).ToString() };
}
yield return new SseItem<string>("", "done");
}
public sealed record ChatRequest(string Question);
On the wire, each update becomes an event like this:
event: delta
data: world
id: 1
Strings are written as they are, and the formatter splits a payload that contains line breaks into several
data: lines, which the client joins back together. Other payload types are serialized as JSON. The result
also sets Cache-Control: no-cache,no-store, turns off response buffering in ASP.NET Core, and sets
Content-Encoding: identity so the compression middleware leaves it alone.
The explicit done event matters more than it looks. EventSource treats the end of a response like a
dropped connection and reconnects, so the client needs a clear signal to call close().
Cancel when the client leaves
In a minimal API, a CancellationToken parameter is bound to HttpContext.RequestAborted, which fires when
the client disconnects. Passing it into GetStreamingResponseAsync means a closed tab cancels the request to
the model provider too, instead of letting your server read an answer nobody will see. Whether the provider
stops generating, and billing, at that moment is up to the provider, but cancelling is the only signal you
can send. The result passes RequestAborted to its writer as well, and [EnumeratorCancellation] lets that
token reach the iterator too.
Errors need a decision of their own. Once the first event is written, the status code is already 200 and
cannot change. A provider failure halfway through should become a final error event the client can show,
not a cut-off stream that looks like a network drop. C# does not allow yield return inside a try block
that has a catch, so that catch has to sit around the call to MoveNextAsync, not around the yield.
Heartbeats and proxy buffering
Two things between your server and the browser can quietly break a stream.
The first is buffering. A reverse proxy that collects the whole response before passing it on turns your
stream back into a spinner. nginx buffers proxied responses by default; the X-Accel-Buffering: no header
in the endpoint above turns that off for one response. Other proxies and gateways have their own settings,
so watch the events arrive one by one through the real path, not just on localhost.
The second is idle timeouts. Proxies and load balancers close connections that have been quiet for a while.
Tokens usually arrive quickly, but there can be long gaps: before the first token on a slow model, or while
an agent waits for a tool call. The usual fix is a heartbeat, an SSE comment line (one starting with a colon)
that the client ignores but that counts as traffic. TypedResults.ServerSentEvents does not send
heartbeats. If your gaps can approach your proxy's timeout, either send your own small ping event from the
iterator, or write the stream with something that does.
What resuming really requires
Last-Event-ID is often described as if it gave you resumable streams. It gives you the client half. The
browser remembers the last id and sends it back; everything else is your job.
For that to work, four things have to be true.
- The generation must outlive the connection. In the endpoint above, a dropped connection cancels the model call, so there is nothing to resume. To resume, the generation runs in the background, keyed by an answer id, and the SSE endpoint only reads what it produced. That trades away cancel-on-disconnect, so decide how long an abandoned generation keeps running.
- The output must be stored. Every token goes into a log with its id, so a reconnecting client can be
served everything after its
Last-Event-ID. Generating again is not a resume: the model will write a different answer. - Any instance must be able to serve it. Behind a load balancer the reconnect can land on another server, so the log can't live only in the memory of the one that started the answer.
- Old logs must expire. Otherwise every answer ever streamed stays in memory or in your cache.
The design also fixes the GET problem: the question goes in a POST, and the stream is a GET that
EventSource can open and reconnect to on its own. When an answer is finished and fully delivered, the
events endpoint can answer a later reconnect with 204 No Content, which tells EventSource to stop trying.
If you don't need resume, don't build it. A chat that restarts a failed answer with a retry button is honest and much simpler.
OrionStream
I maintain OrionStream, an SSE library for ASP.NET Core that covers several of the
pieces above. Its writer sets the SSE headers, including X-Accel-Buffering: no, sends heartbeat comments
on an idle stream, and stops when RequestAborted fires. Its hub publishes events to topics with a bounded
buffer per subscriber, assigns wire ids, and keeps a per-topic replay buffer so a client reconnecting with
Last-Event-ID resumes after the last event it saw. The OrionStream.Redis package moves that replay
backlog to Redis, so resume works across instances and restarts.
For token streams, two settings deserve attention. By default a full subscriber buffer drops its oldest
event, which is right for a live dashboard and wrong for the words of a sentence; size
SubscriberCapacity for your bursts or choose another FullBufferPolicy. And the in-memory replay buffer
keeps a topic's backlog until the process ends, so for one topic per answer use the Redis store, which takes
a backlog time-to-live.
For one browser reading one answer, the built-in TypedResults.ServerSentEvents is enough. I reach for a
library when the same events go to many clients, or when resume has to survive a restart.
