Skip to main content

14 posts tagged with "ai"

View All Tags

Chunking decides your RAG quality before the model does

· 7 min read

When a RAG system gives a bad answer, the first instinct is to blame the model, or to try a bigger one. More often the right text never reached the model at all, because of a decision made long before the question: how the documents were cut into chunks.

A chunk is the unit of everything that follows. It is what gets embedded, what gets ranked and what the model reads. If a chunk holds half a thought, no model will put the other half back.

What I learned building partner-center-mcp

· 8 min read

Building against the Partner Center REST API means living in Microsoft Learn. Microsoft archived the Partner Center .NET SDK in June 2023 and points partners at the REST APIs, but the SDK still turns up in search results, and so does code that requests tokens for the retired graph.windows.net audience and gets a 401 with error 900420 back. A model trained on that material reproduces it with confidence.

partner-center-mcp is the MCP server I wrote to put the current documentation next to the agent instead. This post is about the decisions behind it, and what I would tell anyone designing an MCP server.

Measure retrieval before you tune prompts

· 8 min read

When a RAG system gives a wrong answer, the first thing people reach for is the prompt. Add "only use the context below", add "if you don't know, say so", try another model. Sometimes it helps. Often the real problem happened one step earlier: the chunk with the answer never reached the prompt, and no wording can make a model quote a document it was never shown.

So I split evaluation into two questions and answer them in order. Did retrieval find the right sources? Given those sources, did generation use them well?

Vector search in the Postgres you already run

· 9 min read

When a RAG prototype outgrows its in-memory list, the usual next step is a new piece of infrastructure: a dedicated vector database, with its own deployment, its own backups and a sync job that copies data into it. If you already run Postgres, there is a shorter step. pgvector adds a vector type, distance operators and approximate indexes to the database you already have, and the vectors sit in the same rows as the text they describe.

This post is what I would want to know before putting it in production: the column, the operators, the indexes and their knobs, the filtering trap, and how it looks from .NET.

Prompt injection is an input problem, not a prompt problem

· 8 min read

Prompt injection is usually described as a trick played on the model: someone writes "ignore your previous instructions", and the model obeys. That framing leads to the wrong fix. If the problem is a model being fooled, the answer seems to be a better prompt that tells it not to be fooled.

I think the problem is older and more familiar than that. A system takes text it doesn't control, mixes it with its own instructions, and acts on the result. We have seen that shape before, in SQL injection and in cross-site scripting, and in neither case was the fix a politer request to the database or the browser.

RAG without the magic

· 8 min read

Retrieval-augmented generation sounds like a product category. It is a short pipeline: find the pieces of your own text that look related to a question, paste them into the prompt, and ask the model to answer from them and nothing else.

Frameworks that sell RAG hide those steps behind one call. That is convenient until an answer is wrong and you can't tell which step failed. So here is the whole thing in plain C#, one step at a time, with what each step is for and where it breaks.

Caching LLM calls, and when a semantic cache lies

· 8 min read

LLM calls are slow, billed per token, and a lot of them repeat. The same support question arrives a hundred times a day, the same document gets summarised on every page view. Caching is the obvious fix, and there are two kinds. One is boring and safe. The other saves more and can answer a question nobody asked.

This post is about both: what has to go into an exact-match cache key, why a semantic cache can return the right answer to the wrong question, and how to scope, expire and measure it so you find out before your users do.

Streaming LLM answers to the browser with Server-Sent Events

· 9 min read

A model writes its answer one token at a time. If your API waits for the last token before it returns anything, the user stares at a spinner for the whole generation, even though the first sentence was ready long before.

Streaming fixes that, and in ASP.NET Core it is now a few lines of code. The interesting parts are around those lines: which transport to use, what happens when the user closes the tab, what proxies do to a long response, and what "resume the stream" actually asks of your server.

Getting structured data out of an LLM

· 9 min read

A lot of useful LLM work is not conversation at all. You hand the model an invoice, a contract or an email and you want fields back: a number, a date, a list of lines, something your code can store and add up. The model is good at reading messy documents. It is not, by default, good at handing back something a program can rely on.

The gap between those two is where most extraction code goes wrong. This post walks through closing it in C#, with an invoice as the running example: ask for a shape, check what comes back as if a stranger sent it, retry with a reason, and know when to stop and ask a person.

Designing tools an LLM can use correctly

· 8 min read

When you give a model a tool, you are publishing an API. The consumer is an unusual one, though. It never reads your source, never opens the documentation site, and never asks a colleague what a parameter means. All it gets is a name, a paragraph of description and a JSON Schema, and from those it has to decide whether to call the tool, when, and with what.

Everything I believe about API design still applies. The difference is that the margin for ambiguity is gone: a developer who misreads a parameter finds out in code review, a model finds out in production, if at all.