Skip to main content

RAG without the magic

· 8 min read

Retrieval-augmented generation sounds like a product category. It is a short pipeline: find the pieces of your own text that look related to a question, paste them into the prompt, and ask the model to answer from them and nothing else.

Frameworks that sell RAG hide those steps behind one call. That is convenient until an answer is wrong and you can't tell which step failed. So here is the whole thing in plain C#, one step at a time, with what each step is for and where it breaks.

The problem it solves​

A model knows what was in its training data, frozen at some date, and nothing about your internal documentation, your tickets or your API. You can't paste all of that into every prompt: context windows are large but not unlimited, every token costs money, and models get worse at finding one fact buried in a very long prompt. Fine-tuning teaches style and format better than it teaches facts, and it can never tell you where an answer came from.

RAG takes the other route. The knowledge stays outside the model, your code looks up the relevant part when a question arrives, and the model only sees that part. The model does what it is good at, reading and writing language; the lookup is ordinary code you can test.

The pipeline​

There are two halves, and they run at different times. Indexing happens once, and again whenever the documents change. Answering happens on every question. The one thing the two halves must agree on is the embedding model: vectors from two different models live in different spaces and can't be compared.

I use Microsoft.Extensions.AI for both halves. IEmbeddingGenerator<string, Embedding<float>> turns text into vectors and IChatClient talks to a chat model. OpenAI, Azure OpenAI, Ollama and others have implementations, so nothing below is tied to one provider. The interfaces come from Microsoft.Extensions.AI.Abstractions and the vector math from System.Numerics.Tensors.

Step 1: chunk​

Embedding a whole document gives you one vector that averages everything in it, and a question about one paragraph matches that average only weakly. So documents are split into chunks of a few paragraphs, and each chunk gets its own vector.

How you split matters more than anything else in this post, which is why it has a post of its own. Here I assume each source has already become a list of strings.

Step 2: embed and store​

An embedding is a vector of floats in which texts with similar meaning land close together. "How do I reset my password" and "I forgot my login" share almost no words, and still end up near each other. That is the reason to use embeddings instead of keyword search.

using System.Numerics.Tensors;
using Microsoft.Extensions.AI;

public sealed record Chunk(string Source, string Text, ReadOnlyMemory<float> Vector);

public sealed class InMemoryIndex(IEmbeddingGenerator<string, Embedding<float>> embedder)
{
private readonly List<Chunk> _chunks = [];

public async Task AddAsync(string source, IReadOnlyList<string> texts, CancellationToken ct = default)
{
GeneratedEmbeddings<Embedding<float>> embeddings =
await embedder.GenerateAsync(texts, cancellationToken: ct);

for (int i = 0; i < texts.Count; i++)
_chunks.Add(new Chunk(source, texts[i], embeddings[i].Vector));
}

// SearchAsync comes in step 3.
}

GenerateAsync takes the whole batch, so a document costs one request rather than one per chunk. Providers limit and bill by request, so this matters as soon as there are more than a handful of documents.

Where it fails: the embedding model sees only the chunk. A chunk that says "set it to 30 seconds" without saying what "it" is embeds as something vague, and no question will find it. Embedding models also have an input limit, and longer text is truncated or rejected, depending on the provider. And if you change the embedding model, every stored vector has to be computed again.

Step 3: embed the question and rank​

The question goes through the same embedding model. Then every chunk is scored by cosine similarity, which measures the angle between two vectors: 1 means they point the same way, 0 means they have nothing in common. TensorPrimitives.CosineSimilarity computes it over spans with SIMD.

public async Task<List<(Chunk Chunk, float Score)>> SearchAsync(
string question, int k = 4, CancellationToken ct = default)
{
ReadOnlyMemory<float> q = await embedder.GenerateVectorAsync(question, cancellationToken: ct);

return _chunks
.Select(c => (Chunk: c, Score: TensorPrimitives.CosineSimilarity(q.Span, c.Vector.Span)))
.OrderByDescending(x => x.Score)
.Take(k)
.ToList();
}

This is a linear scan: every question is compared with every chunk. For a documentation folder that is fast enough, and it is the easiest version to debug, because you can print every score.

Where it fails: top-k always returns k results, even when nothing is relevant. Ask about refunds, and the fourth hit may be about shipping, with a score that looks respectable. A minimum score helps, but the right threshold depends on the embedding model and on your data, so pick it by looking at real scores, not by guessing. Similarity is also not the same as relevance. Embeddings are weak at exact strings such as error codes, product names and identifiers, exactly where keyword search is strong. That is why hybrid search, vectors and keywords ranked together, exists.

Step 4: the grounded prompt​

static async Task<string> AskAsync(IChatClient chat, InMemoryIndex index, string question)
{
var hits = await index.SearchAsync(question, k: 4);

var sources = string.Join("\n\n", hits.Select((h, i) =>
$"[{i + 1}] {h.Chunk.Source}\n{h.Chunk.Text}"));

List<ChatMessage> messages =
[
new(ChatRole.System, """
Answer using only the numbered sources. Cite them like [1].
If the sources do not contain the answer, say "I don't know."
"""),
new(ChatRole.User, $"Sources:\n{sources}\n\nQuestion: {question}"),
];

ChatResponse response = await chat.GetResponseAsync(messages);
return response.Text;
}

Three things in that prompt carry the weight. The sources are numbered and named, so the model can cite them and you can show the user where an answer came from, and the user can check it. The model is told to answer only from those sources. And it is explicitly allowed to say "I don't know". Without that permission a model asked a question will answer it, from its training data or from nothing, and the made-up answer sounds exactly as sure as a real one.

Where it fails: instructions are not guarantees. The model can still mix in outside knowledge, or cite [2] for a sentence that came from [3]. If citations matter, check them in code, at least that every cited number exists, and read a sample of answers against their sources.

Retrieved text is also untrusted input. If a document says "ignore your previous instructions", that sentence is now part of your prompt. Retrieval is a prompt injection path, and documents that users can write deserve the same suspicion as anything else users send you.

When to leave the in-memory list​

The list lives in process memory. It is embedded again on every restart, it can't be shared between instances, and every search reads all of it. Move to a real store when one of those starts to hurt, not before.

  • pgvector, if you already run Postgres. Vectors sit next to the rows they describe, in the same transactions and the same backups.
  • Azure AI Search, if you are on Azure and want hybrid keyword and vector search and filtering as a managed service.
  • Qdrant, if you want a dedicated vector database with filtering on stored fields, self-hosted or managed.

All three offer approximate nearest-neighbour indexes such as HNSW, which give up a little recall for a lot of speed. Microsoft.Extensions.VectorData puts a common interface over them if you want to keep the choice open. None of them is magic either: they make the lookup persistent and fast, and they don't make your chunks better or your prompt more honest.

How to know it works​

Without measurement you are tuning by feel. Write down twenty or thirty real questions, each with the source that should answer it, and check two things separately. Did retrieval put the right chunk in the top k? Did the model answer correctly from it?

When an answer is wrong, the first check tells you whether to fix the retrieval or the prompt. Start with retrieval: the model can't use a chunk it never saw. And the most common reason it never saw the right chunk is the way the documents were split, which is where the next post picks up.