Chunking decides your RAG quality before the model does
When a RAG system gives a bad answer, the first instinct is to blame the model, or to try a bigger one. More often the right text never reached the model at all, because of a decision made long before the question: how the documents were cut into chunks.
A chunk is the unit of everything that follows. It is what gets embedded, what gets ranked and what the model reads. If a chunk holds half a thought, no model will put the other half back.
What a good chunk is
A good chunk is about one thing, and it makes sense on its own.
About one thing, so its embedding is sharp and not an average of three topics. Makes sense on its own, because the model reads it without its neighbours and has to be able to use it anyway. Those two pull in opposite directions: the smaller a chunk, the more focused it is and the less context it carries. Every chunking strategy is a way of trading one against the other.
Fixed-size chunking
The simplest strategy is to start a new chunk every N tokens. It works on any text and it makes cost easy to predict. But it cuts wherever the counter runs out: in the middle of a sentence, between a heading and its first paragraph, halfway through a code sample. The signature of a method ends up in one chunk and its body in the next, and neither is useful alone.
The usual patch is overlap: repeat the end of each chunk at the start of the next one, so that a sentence cut at the border appears whole in at least one of them. Overlap helps, but it has a price. It adds storage and embedding calls, and it produces near-duplicate hits that fill the top-k slots with the same text twice. I see heavy overlap as a sign that the boundaries are in the wrong place.
Structure-aware chunking
Most documents developers want to search already have structure: headings, paragraphs, lists, code blocks. The author has already decided where one idea ends and the next begins. Use that.
For markdown, these are the rules I would start with:
- start a new chunk at every heading;
- if a section is too long, split it on paragraph boundaries, the blank lines;
- never split inside a code fence; a code block too big for one chunk becomes a chunk of its own;
- keep a heading with the text under it, because a chunk that is only a heading is noise.
The same idea carries over to other formats. HTML has its heading and pre elements, source code has
functions and classes, a transcript has speaker turns. Fixed-size splitting is the fallback for text that
really has no structure, not the default.
Chunk size
There is no universally right size. Small chunks give precise embeddings and precise matches, but each one carries little context, and an answer may need three of them. Large chunks bring more context per hit, but their embeddings blur across topics, fewer of them fit in the prompt, and the model has more irrelevant text to read past.
The questions matter too. A lookup like "what is the default timeout" is served well by small chunks. "How does the retry policy work" needs a whole section. And the embedding model has an input limit you should stay well under.
So pick a starting point (a few hundred tokens is a common one), then change it based on a set of real questions with known answers, as described in RAG without the magic, and not by feel. Count in tokens rather than characters, because both the embedding limit and the prompt budget are counted in tokens.
Keep metadata on every chunk
A chunk without its source is a quote without attribution. At the least, keep:
- the source document, as a path or URL;
- the heading path, such as
Guide > Install > Windows; - its position in the document;
- whatever you will filter on: product, version, language, who may read it.
That metadata does three jobs. It gives the user a citation they can click. It lets you filter before ranking: only documentation for version 3, only documents this user is allowed to see. Do that filtering in the store as part of the search, not after top-k, or all k results may be filtered away. And never leave access control to the model; a document the user may not read must not reach the prompt at all. The third job is putting chunks back together later, which small-to-big retrieval below depends on.
A markdown-aware chunker
public sealed record MdChunk(string Source, string HeadingPath, string Text);
public static IEnumerable<MdChunk> Chunk(string source, string markdown)
{
var headings = new List<string>();
var buffer = new StringBuilder();
bool inFence = false;
foreach (var line in markdown.ReplaceLineEndings("\n").Split('\n'))
{
if (line.TrimStart().StartsWith("```")) inFence = !inFence;
int level = inFence ? 0 : line.TakeWhile(ch => ch == '#').Count();
if (level is > 0 and <= 6 && line.Length > level && line[level] == ' ')
{
if (Flush() is { } chunk) yield return chunk;
while (headings.Count >= level) headings.RemoveAt(headings.Count - 1);
headings.Add(line[(level + 1)..].Trim());
continue;
}
buffer.AppendLine(line);
}
if (Flush() is { } last) yield return last;
MdChunk? Flush()
{
var text = buffer.ToString().Trim();
buffer.Clear();
return text.Length == 0 ? null : new(source, string.Join(" > ", headings), text);
}
}
The heading path is a stack. A level-2 heading removes everything at level 2 and deeper, then pushes itself,
so ### Windows under ## Install under # Guide becomes Guide > Install > Windows. A fence line flips a
flag, so a # comment inside a bash block is not mistaken for a heading.
It is a sketch. It doesn't split sections that are too long, doesn't know setext headings (the ones
underlined with ===), and doesn't skip front matter. For real documents I would walk the syntax tree from a
markdown parser such as Markdig instead of reading lines.
Contextual headers
Take a chunk under Install > Windows that says "Run the installer as administrator." Embedded alone, the
word Windows never appears in it, and a question about installing on Windows matches it only weakly. The
information was in the document; the chunker threw it away.
The fix is cheap: put a short header in front of the text before embedding it.
static string TextToEmbed(MdChunk c) => $"{c.Source}\n{c.HeadingPath}\n\n{c.Text}";
Show the same header to the model in the prompt, too, so it knows where each source sits. You can go further and have a model write a sentence or two of context for every chunk, describing where it fits in the document. That costs a model call per chunk at indexing time. The heading path costs nothing, so try it first.
Small-to-big retrieval
The size trade-off has a way out: use one size for searching and another for reading. Index small chunks, a paragraph each, for precise matching, and give each one the id of the section it came from. At question time, rank the small chunks, then hand the model the parent sections they belong to.
public sealed record SmallChunk(string ParentId, string Text, ReadOnlyMemory<float> Vector);
// hits: the top-k small chunks; sections: parent id -> full section text
var context = hits
.Select(h => h.Chunk.ParentId)
.Distinct()
.Select(id => sections[id]);
Two matches in the same section become one parent, which is what you want. The price is size: parents are bigger, so fewer of them fit in the prompt. A middle ground is to return each small chunk together with its immediate neighbours instead of the whole section.
Where I would start
Split on headings, keep the heading path on every chunk and prepend it before embedding, and build the set of test questions before tuning anything. Most of what goes wrong will show up there. Reach for small-to-big when sections turn out too long to embed well, and for generated context when heading paths are not enough. Change one thing at a time, and run the same questions after every change.
