Vectors are not enough: hybrid search and reranking
Most RAG tutorials have the same retrieval step: embed the question, ask a vector store for the nearest chunks, put them in the prompt. It works well on the demo questions, which tend to be phrased like the documents. Then someone pastes an error code into the box and gets five chunks about errors in general, none of them about that code.
That isn't a bad embedding model. It is what embedding search is for. This post is about the two pieces I would add before tuning anything else: keyword search next to the vectors, fused by rank, and a reranker over the result.
What embeddings are good at, and what they lose
An embedding model turns a chunk of text into one point in a space where similar meanings sit close together. That is exactly why it finds "how do I get my money back" when the document says "refund policy". No keyword search does that on its own.
The same compression is where exact terms get lost. A whole chunk becomes a few hundred or a few thousand numbers, and one rare token inside it barely moves the point. Some cases fail again and again:
- Error codes and IDs.
ERR_4012andERR_4013mean different things to you and almost the same thing to the model. An order number or a ticket ID has no meaning to embed at all. - Product and library names. A name the model rarely saw in training is split into sub-word pieces and lands near whatever those pieces resemble.
- Version numbers and flags. "Fixed in 3.2.1" and "fixed in 3.1.2" look the same to cosine similarity.
- Short queries. A two-word query carries little meaning, so its nearest neighbours are close to random among documents on the same topic.
People who type an exact term usually want the document that contains exactly that term. Embedding search can't promise that.
Keyword search still does its job
Full-text search works the other way around. It indexes the tokens themselves, and a ranking function
such as BM25 scores a document by how often the query terms appear in it, how rare those terms are across
the whole collection, and how long the document is. A rare term like ERR_4012 gets a high weight
precisely because it is rare. If a chunk contains it, that chunk comes up.
What full-text search can't do is match "money back" to "refund". So the two are not rivals: each one is strong where the other is weak. Run both and you get the exact matches and the paraphrases.
The hard part is combining them. Cosine similarity lives roughly between -1 and 1, and BM25 scores are unbounded and depend on the collection. Adding them, or normalising both to 0..1 and averaging, means picking weights that change whenever the corpus does.
Reciprocal Rank Fusion
Reciprocal Rank Fusion avoids the problem by ignoring scores altogether and using only positions. Each retriever returns a ranked list, and a document's fused score is the sum over all lists it appears in:
RRF(d) = Σ 1 / (k + rank_r(d))
r
rank_r(d) is the document's 1-based position in list r, and k is a constant; 60 is the value from
the original paper and still the usual default. A document that both retrievers put near the top beats a
document that only one of them loves. A large k flattens the difference between first and fifth place,
so no single list dominates.
The whole algorithm fits in a few lines:
public static class Rrf
{
public static List<(string Id, double Score)> Fuse(
IEnumerable<IReadOnlyList<string>> rankings, int k = 60)
{
var scores = new Dictionary<string, double>();
foreach (var ranking in rankings)
{
for (int i = 0; i < ranking.Count; i++)
{
// ranks are 1-based: the first result contributes 1 / (k + 1)
scores[ranking[i]] = scores.GetValueOrDefault(ranking[i]) + 1.0 / (k + i + 1);
}
}
return scores
.OrderByDescending(p => p.Value)
.Select(p => (Id: p.Key, Score: p.Value))
.ToList();
}
}
Using it is one call:
IReadOnlyList<string> vectorHits = await vectorIndex.SearchAsync(question, top: 50);
IReadOnlyList<string> keywordHits = await textIndex.SearchAsync(question, top: 50);
var candidates = Rrf.Fuse([vectorHits, keywordHits]).Take(50).ToList();
(vectorIndex and textIndex stand in for whatever you search with; all RRF needs from them is IDs in
order.)
Two things are worth knowing. RRF throws the scores away, so it can't tell "the best of a bad lot" from a real match; it only orders. And it adds up evidence, so a document found by both retrievers gets a real boost. That second property is usually what you want.
Rerank the top of the list
Hybrid retrieval improves recall: the right chunk is more likely to be somewhere in the top 50. It doesn't make it first. Both retrievers compare a query and a document that were encoded separately: the embedding of the document was computed long before anyone asked the question, and BM25 only counts tokens.
A cross-encoder reranker reads the query and the document together, as one input, and outputs a single
relevance score. Because it sees both at once, it can notice that the document mentions ERR_4012 only
in a list of codes that are unrelated, or that it answers a slightly different question. That makes it
much more accurate than either retriever, and much more expensive: nothing can be computed ahead of time,
and every query-document pair is a model call.
So it goes at the end, over a short list:
The sizes are knobs, not rules. The candidate list has to be long enough that the right chunk is usually in it, and short enough that reranking stays within your latency budget. How many chunks end up in the prompt depends on chunk size and on the model's context. I would tune each of these against measurements, not by feel.
Rerankers come as hosted APIs and as open models you can run yourself; the open cross-encoders trained on MS MARCO are a common starting point. Unlike RRF, a reranker's score is comparable across documents for the same query, so you can also use it to drop candidates that fall below a threshold instead of always filling the prompt with five chunks.
You probably don't have to build it
Hybrid search is common enough now that most stores offer some form of it:
- Postgres: pgvector for embeddings and the built-in
tsvectorfull-text search live in the same database, so both searches and the fusion can be one SQL query. Note that Postgres ranks full-text matches withts_rank, which isn't BM25, but plays the same role here. - Azure AI Search runs a vector query and a full-text query in one request and merges them with RRF, and its semantic ranker can rerank the top of the result.
- Elasticsearch, OpenSearch, Qdrant and Weaviate each have their own way to combine vector and keyword results.
What I would look for is not which one has the feature, but whether you can see both lists before they are fused. When a question returns the wrong chunk, the first thing to ask is which retriever missed it, and you can only answer that if you can see each one on its own.
The order I would do it in
- Keep vector search, and add keyword search next to it.
- Fuse them with RRF,
k = 60, and leave it there until a measurement says otherwise. - Put a cross-encoder over the fused top N.
- Only then look at the prompt.
Each step should earn its place with numbers on your own questions, not mine or a vendor's. Measure retrieval before you tune prompts is about how to get those numbers cheaply.
