Search relevance tuning: a disciplined first pass

Most relevance work fails the same way: someone tweaks a boost, eyeballs three queries, ships it, and next quarter nobody knows which of the accumulated tweaks still helps. The fix is not a smarter ranking model — it is measurement. Here is the first pass we run on product and site search, in order, on Elasticsearch or OpenSearch.

Step 1: Build a judgment list

Pull your top queries from analytics — the head 50-200 queries typically cover a large share of traffic — and add a sample of known-difficult ones: misspellings, part numbers, synonyms your users actually type. For each query, record which documents are good results. Product owners and support staff are excellent judges; a spreadsheet with query, doc_id, rating on a 0-3 scale is entirely sufficient. This list is the asset. Everything else on this page is replaceable; the judgment list is what turns opinion into engineering.

Step 2: Pick a metric and wire up rank_eval

Elasticsearch and OpenSearch both ship a ranking evaluation API that scores a batch of rated queries against the live index:

GET products/_rank_eval
{
  "requests": [ { "id": "q1", "request": { "query": { ... } },
                  "ratings": [ { "_id": "doc-42", "rating": 3 } ] } ],
  "metric": { "dcg": { "k": 10, "normalize": true } }
}

Normalized DCG@10 is a sensible default: it rewards putting the best documents near the top without obsessing over exact order. Script it, run it against every candidate change, and record the score. From here on, no change ships without a before/after number.

Step 3: Fix analysis before touching queries

Analyzer and mapping problems put a ceiling on everything downstream, and they are where the cheap wins live:

• Language analysis — stemming appropriate to your language(s), so "running shoes" matches "running shoe".
• Synonyms — your users' vocabulary mapped to your catalog's ("hoodie" vs "hooded sweatshirt"). Maintain the list as data, review it quarterly.
• Multi-fields — index the same source field several ways: analyzed for full-text, keyword for exact matching and aggregations, possibly an edge-ngram or search_as_you_type field for prefix behavior. Query the right sub-field per purpose instead of forcing one analyzer to do everything.
• Tokenization traps — SKUs, model numbers, and hyphenated terms often need a dedicated pattern; the standard analyzer happily splits "X-100" into tokens nobody searches.

Re-run the evaluation after each fix. It is common for analysis work alone to move the metric more than any amount of subsequent boost-fiddling.

Step 4: Structure the query deliberately

Now, and only now, the query DSL:

• multi_match with explicit field weights — e.g. fields: ["name^3", "brand^2", "description"]. Title-ish fields deserve their boost; body text is context, not the signal.
• minimum_should_match — for multi-term queries, requiring most terms (say "2<75%") cuts the long tail of one-word-matched noise dramatically.
• Phrase and proximity boosts — a match_phrase clause with a slop as a should-clause rewards documents where the terms appear together, which is usually what the user meant.
• Filters are not queries — category, availability, and price constraints belong in filter context: no scoring cost, cacheable, and they stop polluting relevance.

Step 5: Add business signals — gently

function_score (or rank_feature fields) folds in popularity, freshness, or margin. Two disciplines keep this honest: keep the text score dominant (business signals tie-break, they don't override), and re-run the judgment list after every weight change — business boosts are the classic way to win the metric's head queries while quietly wrecking the tail.

What about vector search?

Later, and by measurement. Dense-vector retrieval genuinely helps with vocabulary mismatch — queries where users describe intent in words your catalog never uses. But it adds an embedding pipeline, kNN index memory, and a score-fusion problem, and on catalogs with strong exact-match expectations a well-tuned BM25 setup regularly holds its own. Run the fundamentals first; if the judgment list still shows a class of semantic misses, that is your evidence that hybrid retrieval will pay for its complexity — and you'll have the harness to prove whether it did.

The habit that outlives the pass

Schedule the evaluation to run weekly and after every index or query change. Relevance decays as catalogs and vocabularies drift; a standing metric is the difference between noticing and being told by a customer. The first pass takes days, not months — and it leaves your team with a measuring stick, which is worth more than any single tuning session.