What goes in the context window
Everything a model knows about the current request arrives as one block of input: your instructions, the tool definitions, the documents you retrieved, the conversation so far and whatever the last tool returned. The model has no other memory. If something is not in that block, it does not exist for this call.
Context windows are now large enough that it is tempting to put everything in and let the model sort it out. I think that is the wrong default. The window is a budget, and like any budget it is easier to spend well when you decide what each part is for.
Who competes for the space
A typical request to an agent or a chat assistant carries five kinds of content:
- The system prompt: who the assistant is, what it may and may not do, the output format.
- Tool definitions: the name, description and JSON schema of every tool the model may call. They are sent on every request, whether the model uses them or not.
- Retrieved documents: the chunks your search found for this question.
- Conversation history: earlier questions and answers, including the model's own tool calls.
- Tool results: what those calls returned. Often the largest item, and the least predictable.
Each of these grows for its own reasons, usually written by a different person in a different part of the code. Nobody sees the total until a request fails because the window is full, or until the invoice arrives.
Long context is not free
Size costs in three ways.
Money. Input tokens are billed on every request. With most chat APIs the whole conversation is sent again on each turn, so turn twenty pays for the nineteen before it, and every tool result that is still in the history is paid for again and again.
Latency. The model has to process the entire input before it writes the first output token. A longer prompt means a longer wait before anything appears, which is exactly the delay that streaming is meant to hide.
Attention. A model that accepts a very long input does not use all of it equally well. The paper Lost in the Middle measured models doing best when the relevant information was at the beginning or the end of the context, and worse when it sat in the middle. Newer models handle long inputs better, but I would not bet a correct answer on the model finding one sentence in the middle of a pile of tool output. The maximum window size is a limit, not a target.
Measure each part first
Before trimming anything, find out where the tokens go. The provider reports the total input tokens in the
response, which in Microsoft.Extensions.AI is response.Usage. The total does not tell you which part to
cut, so I also count each part on my side:
using System.Text.Json;
using Microsoft.Extensions.AI;
using Microsoft.ML.Tokenizers;
Tokenizer tok = TiktokenTokenizer.CreateForModel("gpt-4o");
var report = new Dictionary<string, int>
{
["system"] = tok.CountTokens(systemPrompt),
["tools"] = tools.OfType<AIFunction>()
.Sum(f => tok.CountTokens($"{f.Name} {f.Description} {f.JsonSchema}")),
["retrieved"] = chunks.Sum(c => tok.CountTokens(c)),
["history"] = history.Where(m => m.Role != ChatRole.Tool).Sum(m => CountTokens(tok, m)),
["tool results"] = history.Where(m => m.Role == ChatRole.Tool).Sum(m => CountTokens(tok, m)),
};
static int CountTokens(Tokenizer tok, ChatMessage m) => m.Contents.Sum(c => c switch
{
TextContent t => tok.CountTokens(t.Text),
FunctionCallContent call => tok.CountTokens(call.Name + JsonSerializer.Serialize(call.Arguments)),
FunctionResultContent r => tok.CountTokens(r.Result?.ToString() ?? ""),
_ => 0,
});
The tokenizer comes from Microsoft.ML.Tokenizers with its O200kBase data package. It matches one model
family exactly and the others only roughly, and providers add formatting tokens of their own, so treat the
numbers as estimates. They are good enough to show which part is growing. Log them per request next to the
provider's real total, and the first time you look at a long conversation you will probably know what to fix.
Stable things first
Order matters for two reasons. Instructions at the start set the frame for everything after them. And it decides whether prompt caching works for you.
Major providers can reuse the work they did on the beginning of a previous prompt, but only for an exact prefix: the cached part runs from the first token up to the first difference. Some providers do this automatically, others need you to mark where the cacheable part ends; there is a minimum length, and the cache expires after some minutes without use. A hit makes those input tokens cheaper and shortens the time to first token. The OpenAI and Anthropic documentation both state that tool definitions are part of that prefix, so adding, removing or reordering tools changes it.
So: tools and the system prompt first, and identical byte for byte across requests. No timestamp, user name or request id in the system prompt; those go into the latest message. History next, growing at the end. Retrieved documents and the new question last.
There is a tension with the next rule. Every time you trim or summarise history, you change the prefix from that point on. Trimming in larger steps, rather than dropping one message every turn, keeps the prefix stable between cuts.
Trim history by turns, and summarise what you drop
Keeping only recent messages is the simplest budget, but a cut in the wrong place breaks the request. A tool result without the tool call that produced it is invalid input for some providers, and confusing for all of them. So I cut only where a user turn begins:
static List<ChatMessage> KeepRecentTurns(
List<ChatMessage> history, Func<ChatMessage, int> count, int budget)
{
int used = 0, start = history.Count;
for (int i = history.Count - 1; i >= 0; i--)
{
used += count(history[i]);
if (used > budget) break;
if (history[i].Role == ChatRole.User) start = i; // cut only where a turn begins
}
return history.GetRange(start, history.Count - start);
}
What falls off the front can be summarised by a cheaper model into a short paragraph that sits before the
recent turns. Microsoft.Extensions.AI has experimental reducers for this, MessageCountingChatReducer and
SummarizingChatReducer. They count messages rather than tokens, which is a weaker budget, but they are a
reasonable start.
A summary loses detail, and you don't control which detail. Facts the conversation depends on, such as the order id being discussed, a decision the user confirmed or a constraint they stated, are better kept as explicit state that your code writes into the prompt than left to a summary to remember.
Truncate tool output, and say so
One tool call that returns a whole log file or ten thousand rows can take more space than everything else together. Cap it, and tell the model that you did:
static string Truncate(string output, int maxChars) =>
output.Length <= maxChars
? output
: output[..maxChars] +
$"\n[Output truncated: showing the first {maxChars} of {output.Length} characters. " +
"Narrow the query or ask for the next page to see more.]";
Without the note the model treats a partial list as complete and answers confidently from it. With the note it can ask for the rest. Better still is a tool that returns less in the first place, with filters and paging, which I wrote about in designing tools an LLM can use.
Retrieve instead of stuffing
If the answer lives in a large set of documents, putting all of them in the prompt "to be safe" pays the full price on every request and buries the relevant paragraph in the middle. Search for the few chunks that matter and send only those; that is the whole idea behind RAG. Retrieved text also deserves a smaller budget than people give it. I would rather send four good chunks than twenty mediocre ones, and how many your data needs is something to measure, not guess.
Keep the tool list short
Every tool definition is paid for on every request, and every extra tool is one more option the model can pick wrongly. An agent with sixty tools spends a noticeable part of its window describing things it will not use in this conversation. Fewer, broader tools help, and so does offering only the tools that make sense for the current task.
Choose that set once per task, not once per message. Tool definitions are part of the cached prefix, so a tool list that changes between turns invalidates the cache for everything after the change.
None of these rules is clever. They are what you would do with any scarce resource: know where it goes, put the stable parts where they can be reused, and stop paying repeatedly for things that no longer help.
