Every quarter someone asks us to "add AI search" to a catalog that is running plain match queries against a text field. The honest answer is that vector search is not an upgrade path from BM25; it is a second retriever that fails differently. BM25 misses when the user's words are not the document's words. Dense retrieval misses when the query is an exact identifier, a part number, or a rare token it never saw in training. Hybrid search runs both and merges the results, and it is usually the right first move — but only if you can show the merge beat the baseline.
This is how we implement it, and how we decide whether to keep it.
Step 0: fix the lexical baseline first
Do not benchmark vectors against a badly tuned match query. If your current search has no analyzer work, no field boosting, and no synonym handling, the cheap wins are still on the table and they cost no GPU. Get a defensible BM25 baseline — multi_match with type: best_fields, sensible boosts on title versus body, an edge n-gram or search_as_you_type field if you need prefix behavior — and record its scores. Everything below is measured against that number, not against nothing.
Step 1: mappings
On Elasticsearch, add a dense_vector field alongside the existing text fields. Do not create a separate index; you want one document, both representations, so filters and business rules apply once.
PUT products/_mapping
{
"properties": {
"description_vector": {
"type": "dense_vector",
"dims": 384,
"index": true,
"similarity": "cosine",
"index_options": { "type": "int8_hnsw", "m": 16, "ef_construction": 100 }
}
}
}
On OpenSearch the equivalent is knn_vector with the k-NN plugin, and the index needs index.knn: true:
PUT products/_mapping
{
"properties": {
"description_vector": {
"type": "knn_vector",
"dimension": 384,
"method": { "name": "hnsw", "engine": "lucene", "space_type": "cosinesimil" }
}
}
}
Three decisions hide in there:
- Dimensions. 384-dim models (MiniLM class) are usually enough for product and doc search and cost a third of a 1024-dim model in RAM and disk. Start small; the retrieval gap is often smaller than the bill difference.
- Quantization.
int8_hnswstores 8-bit values and keeps full-precision vectors on disk for rescoring. It cuts the in-memory graph roughly 4x for a recall loss that is typically under a point. On a 5-million-document index this is the difference between a vector workload that fits in node RAM and one that does not. m/ef_construction. These control HNSW graph quality versus build time.m: 16,ef_construction: 100is a reasonable default. Raising them improves recall and slows indexing; it is not where your relevance problem lives.
Step 2: get vectors into documents
Two options, and the trade-off is about who owns the model.
Embed in your own pipeline. Your indexer calls an embedding service, writes the array into the document. More moving parts, but you control model versions, batching, and cost, and you can re-embed on your own schedule.
Embed in the cluster. Elasticsearch can host a model and embed at ingest time via an inference processor, or you can skip the plumbing entirely with semantic_text, which handles chunking and embedding on write and query. OpenSearch has neural search with an ingest processor and a model connector to a hosted provider. Less code; the cluster now owns ML work and its capacity planning.
Whichever you pick, write down the model name and version in the index metadata. The single most common vector outage we see is queries embedded with a different model than the documents. There is no error for that — relevance just goes quietly random.
And re-embedding is a reindex. Budget it: at a few thousand docs per second and an embedding call per doc, a 20-million-document catalog is a multi-hour job you want rehearsed against an alias swap, not run live.
Step 3: run both retrievers and fuse them
The naive approach is to add the kNN score to the BM25 score. Don't. The two are on incomparable scales — BM25 is unbounded and corpus-dependent, cosine similarity sits in a narrow band — so any fixed weight is a magic number that breaks when your corpus grows.
Use reciprocal rank fusion. RRF ignores scores and uses rank position: each document scores sum(1 / (k + rank)) across result lists, with k (commonly 60) damping the top-heavy weighting. It has one tunable, it is scale-free, and it is hard to make worse.
In Elasticsearch 8.x/9.x this is a retriever:
POST products/_search
{
"retriever": {
"rrf": {
"retrievers": [
{ "standard": { "query": { "multi_match": {
"query": "waterproof hiking boots",
"fields": ["title^3", "description"] } } } },
{ "knn": { "field": "description_vector",
"query_vector_builder": { "text_embedding": {
"model_id": "my-embedding-model",
"model_text": "waterproof hiking boots" } },
"k": 50, "num_candidates": 200 } }
],
"rank_window_size": 50,
"rank_constant": 60
}
},
"size": 20
}
On OpenSearch, the same idea is a search pipeline with a normalization processor (min_max normalization plus arithmetic or harmonic combination) over a hybrid query containing the two clauses. Different syntax, same job: put both lists on a common footing before merging.
Two parameters to actually think about:
num_candidatesis how many nodes HNSW explores per shard before returningk. It is your recall/latency dial.num_candidatesof 2–5xkis a normal starting range; going much higher buys little recall and costs real milliseconds.rank_window_sizemust be at least as large as the page you serve. If you return 20 results from a fusion window of 10, half your page is coming from one retriever by accident.
Step 4: filters, because business rules do not care about embeddings
A vector query with a post-filter can return an empty page: HNSW finds 50 nearest neighbours, your in_stock: true filter deletes 48 of them. Use the filter inside the kNN clause so the graph search is constrained during traversal, and apply the same filter to the lexical retriever. Otherwise the two lists disagree about what is eligible and RRF happily promotes a product you cannot sell.
Step 5: measure, or you are guessing
This is the part teams skip, and it is the only part that decides whether hybrid stays.
- Build a judgment set. 100–200 real queries from your logs, weighted toward head traffic, with the top ~10 candidates per query rated 0–3 for relevance. Two raters, disagreements discussed. A weekend of work; it pays for itself for years.
- Score offline. Use the Rank Evaluation API (
_rank_eval) with nDCG@10 and recall@50. Run it against three configurations: BM25 only, kNN only, RRF. You need all three — if kNN alone is worse everywhere and RRF is a wash, you have added infrastructure for nothing. If BM25 wins on head queries and kNN wins on long-tail phrasing, that is the textbook case for keeping the fusion. - Then check online. Offline metrics tell you retrieval got better; only click-through, zero-result rate, and add-to-cart tell you users noticed. Watch the zero-result rate especially — the clearest, most defensible win from hybrid search is usually queries that previously returned nothing.
What it costs
Be honest with your capacity plan before you commit:
- RAM. HNSW graphs want to be resident. A rough figure for float32 is
dims * 4 bytes * docsplus graph overhead; 5M docs at 384 dims is around 7–8 GB per replica before quantization, roughly 2 GB withint8. That is a node-sizing conversation, not a rounding error. - Latency. Two retrievers plus fusion, and if you embed the query in-cluster, an inference hop on every search. Query embedding is frequently the largest single term in hybrid latency. Cache embeddings for head queries; they repeat far more than you expect.
- Indexing throughput. Graph construction is CPU work on the write path. Expect a measurable drop in indexing rate and re-check your bulk sizing after enabling it.
The short version
Tune BM25 first. Add one dense_vector or knn_vector field to the existing index, quantized, at the smallest dimension that works. Fuse with RRF rather than hand-weighted score addition. Filter inside the kNN clause. Then prove it with a judgment set and _rank_eval before anyone calls it done.
If hybrid retrieval only ties your tuned lexical baseline, that is a legitimate result and worth knowing for the cost of a week. Most catalogs we test land somewhere in between: no change on head queries, a real drop in zero-result rate on the tail. Whether that justifies the RAM depends on what those tail queries are worth to you.
If you are sizing this on a live cluster and want a second opinion on the mapping, the node budget, or the measurement plan, get in touch.