When a RAG feature gives bad answers, the first instinct is to blame the model or rewrite the prompt. In most of the clusters we look at, the model never saw the right passage. The retrieval step returned ten chunks, none of which contained the fact, and the LLM did what LLMs do with insufficient context. That is a search problem, and it is measurable with the same tools you already use for product search.
This post is about the retrieval half: how to chunk, how to compose the query, when a reranker is worth its latency, and how to know whether any change helped. It applies to Elasticsearch 8.x/9.x and, with different syntax, to OpenSearch.
Measure recall before you tune anything
Build a small evaluation set first: 50-100 real questions, each annotated with the chunk (or document) that actually contains the answer. Support tickets and internal search logs are the cheapest source. Then measure recall@k — in what fraction of questions does the right chunk appear anywhere in the top k you pass to the model?
That single number splits the problem cleanly:
- Recall@10 is 60% → retrieval is the bottleneck. Prompt work is wasted effort.
- Recall@10 is 95% and answers are still wrong → look at ordering, context budget, or the prompt.
The ranking evaluation API scores this directly:
GET kb-chunks/_rank_eval
{
"requests": [
{ "id": "q-114",
"request": { "query": { "match": { "text": "rotate the API key for a service account" } } },
"ratings": [ { "_id": "doc-8821#3", "rating": 1 } ] }
],
"metric": { "recall": { "k": 10 } }
}
For RAG, recall@k matters more than nDCG. The model reads all k passages; it does not care much whether the right one was ranked second or seventh. It cares a great deal whether it was there at all.
Chunking: match the unit of retrieval to the unit of answer
Chunking is the decision with the largest effect on recall, and it is almost always made once, arbitrarily, in a loader script.
Starting points that hold up:
- Split on structure, not on character count. Markdown headings, HTML sections, docstrings, ticket fields. A chunk that starts mid-sentence because it hit 512 characters embeds badly and reads badly.
- Roughly 200-500 tokens per chunk for prose knowledge bases. Smaller chunks give sharper embeddings and worse context; larger chunks dilute the embedding until it matches everything weakly.
- Overlap 10-20%, or carry the parent heading into each chunk's text. Facts that straddle a boundary are the single most common cause of a missing answer.
- Keep a
doc_idandordinalon each chunk so you can expand to neighbours at query time — retrieve the chunk, send the chunk plus its two neighbours to the model. This gets you small-chunk precision with large-chunk context, and it costs one extra term query.
In Elasticsearch, semantic_text will chunk and embed for you at ingest; that is the right default when your documents are reasonably uniform prose. Do your own chunking when structure matters — code, tables, policy documents with numbered clauses — because a generic splitter will cut them in the wrong places.
One mapping note: store the chunks as nested objects under the parent document, or as separate chunk documents with a doc_id field. The nested approach keeps parent metadata for filtering; the flat approach is simpler to reindex. Both work. Pick one before you have 40 million chunks, because changing it later is a full reindex.
Composing the query: retrievers
RAG queries are hybrid queries. Users ask in their own words (semantic retrieval helps) but also quote error strings, flag names, and part numbers verbatim (lexical retrieval is the only thing that finds those). The retrievers syntax composes both without application-side merging:
GET kb-chunks/_search
{
"retriever": {
"rrf": {
"retrievers": [
{ "standard": { "query": { "match": { "text": "rotate api key service account" } } } },
{ "knn": { "field": "text_embedding", "query_vector_builder": {
"text_embedding": { "model_id": ".multilingual-e5-small", "model_text": "rotate api key service account" } },
"k": 50, "num_candidates": 200 } }
],
"rank_window_size": 50,
"rank_constant": 20
}
},
"size": 50
}
Two practical points. First, filters still apply: tenant ID, product version, and ACL filters belong inside each retriever, and on the kNN side a restrictive filter interacts with num_candidates — too few candidates plus a tight filter returns a short list, silently. Second, retrieve more than you send. size: 50 here, with the top 5-10 going to the model after reranking.
On OpenSearch, the equivalent is a hybrid query with a normalization-processor search pipeline. Same shape, different configuration surface, and the score-normalization behaviour differs enough that you should re-tune rather than port weights.
Reranking: pay the latency where it buys recall
A cross-encoder reranker reads the query and passage together and scores the pair. It is markedly more accurate than either BM25 or a bi-encoder embedding, and markedly more expensive — it runs a model inference per candidate instead of a vector comparison.
So rerank a short list. Retrieve 50 with RRF, rerank those 50, send the top 5:
"retriever": {
"text_similarity_reranker": {
"retriever": { "rrf": { "...": "as above" } },
"field": "text",
"inference_id": "my-rerank-endpoint",
"inference_text": "rotate api key service account",
"rank_window_size": 50
}
}
Budget it honestly. Reranking 50 passages against a hosted rerank endpoint typically adds somewhere in the 100-400 ms range end to end; locally deployed models on ML nodes vary with hardware and batch size. Measure on your own data and your own passage lengths — long chunks cost more, because the cost scales with tokens, not documents.
When it is worth it:
- Yes, when recall@50 is high but recall@5 is not. The right passage is in the candidate set and just ranked badly. This is the common case, and reranking is the cheapest fix available.
- Probably not, when recall@50 is already poor. The reranker can only reorder what retrieval found. Fix chunking and the hybrid query first.
- It depends, for interactive chat where the budget is tight. The model's own generation latency usually dwarfs 200 ms of reranking, so the trade is better than it looks — but if you stream tokens, time-to-first-token is the number your users feel.
Context budget: more passages is not better
It is tempting to send 20 passages because the model's context window allows it. Don't, by default. Longer contexts cost more per call, add latency, and degrade precision — models reliably use the beginning and end of a long context better than the middle. Start at 5 passages, raise it only if your evaluation set shows answers improving, and dedupe near-identical chunks before assembling the prompt. Near-duplicates are endemic in knowledge bases where the same paragraph appears in six documents, and they waste slots that could have held a different fact.
A working order of operations
- Build the 50-100 question evaluation set with known-good chunks.
- Measure recall@10 and recall@50 with the current pipeline. Record them.
- Fix chunking — structure-aware splits, overlap, parent heading in the text. Re-measure.
- Move to a hybrid retriever (BM25 + kNN via RRF). Re-measure.
- Add a reranker over the top 50 if recall@50 is much better than recall@10. Re-measure.
- Tune the number of passages sent to the model against answer quality, not intuition.
Each step produces a number you can show. That matters beyond engineering hygiene: RAG projects get cancelled because nobody could say whether they were improving, and "recall@10 went from 61% to 93%" is a sentence that keeps a project funded.
If you are standing up retrieval for an LLM feature on an existing Elasticsearch or OpenSearch cluster — or the one you built is returning the wrong passages and you need to find out why — that is work we do. Describe the setup through the contact form and we will tell you where we would start.