+1 (806) 515-4974
QueryCrunch

Search latency triage: finding the bottleneck before you add nodes

The usual sequence: search felt fine for a year, then the p99 drifted from 80 ms to 900 ms, and the first proposal on the table is more nodes. Sometimes more nodes is the answer. More often it is the most expensive way to hide a query that reads ten times more data than it needs to. Adding hardware to an unmeasured latency problem also destroys the evidence, because the symptom moves and the cause does not.

Here is the order we work through it, on Elasticsearch or OpenSearch. Each step either names the bottleneck or rules out a whole class of cause, and every step produces a number you can write down.

Step 0: decide which latency you are fixing

Three different numbers get called "search latency":

  • took — milliseconds the cluster spent on the query, coordinating node included. This is what the cluster controls.
  • Client-observed time — took plus serialization, network, connection setup, and any queuing in your application's HTTP client pool.
  • User-observed time — the above plus everything your own service does around the search call.

Compare took against client-observed time for the same requests before anything else. If took is 40 ms and the client sees 700 ms, the cluster is not your problem: look at connection pool size, keep-alive, DNS, TLS handshakes per request, and whether you are running searches sequentially that could be one _msearch. We have watched teams buy nodes for a latency problem that lived entirely in an HTTP client with two connections.

Also fix the statistic. Averages hide everything that matters here. Track p50, p95, p99 per query type — one heavy dashboard query firing every minute can own your p99 while the average looks calm.

Step 1: turn on the slow log and let it name the queries

The search slow log records the actual query body, per shard, over a threshold. Set it on the indices you care about:

PUT my-index/_settings
{
  "index.search.slowlog.threshold.query.warn": "2s",
  "index.search.slowlog.threshold.query.info": "500ms",
  "index.search.slowlog.threshold.fetch.warn": "1s"
}

Two things to understand about it:

  1. Thresholds are per shard, not per request. A request whose shards each take 300 ms may be slow to the user and invisible at a 500 ms threshold. Start lower than you think, then raise it once the noise is characterized.
  2. Query phase and fetch phase are separate. Slow query phase points at matching, scoring, and aggregation work. Slow fetch phase points at large _source documents, huge size values, many highlighted fields, or deep pagination.

Leave it on for a day and read what it caught. In practice a handful of query shapes account for nearly all of the tail, and the slow log hands you their exact bodies to replay.

Step 2: profile the worst query

Take the worst body from the slow log and run it through the profiler:

GET my-index/_search
{
  "profile": true,
  "query": { ... }
}

The output is verbose and worth reading properly. What to look for:

  • Which query clause owns the time. Per-clause timings break down into create_weight, build_scorer, next_doc, score. A wildcard, regexp, or leading-wildcard clause with an enormous next_doc cost is the classic finding.
  • Aggregation time vs. query time. The aggregations section is separate. High-cardinality terms aggs and nested agg trees frequently cost several times the query itself.
  • Shard skew. Profile output is per shard. If one shard takes 600 ms and the others take 30 ms, you do not have a query problem, you have a distribution problem — an oversized shard, a hot node, or a routing key that concentrates documents.

For query-phase cost, the fixes are usually structural rather than clever:

  • Move every non-scoring constraint into filter context. Filters skip scoring and are cacheable; the same clause in must is neither.
  • Replace leading wildcards with what you actually need — an edge_ngram field, search_as_you_type, or a wildcard-typed field. *term* over a large index will never be fast.
  • Bound aggregations. A terms agg with size: 10000 over a million-cardinality field is doing work nobody reads; composite paging or a smaller size is almost always the intent.
  • Kill deep pagination. from: 10000 makes every shard build and sort 10,000 + size hits. Use search_after with a tiebreaker, or a point-in-time reader for stable iteration.
  • Trim the fetch. _source filtering, fewer highlighted fields, and a sane size fix slow fetch phases directly.

Step 3: check whether queries are waiting rather than running

If per-shard times look fine but end-to-end latency is bad, requests are queuing:

GET _cat/thread_pool/search,search_worker,write?v&h=node_name,name,active,queue,rejected
GET _nodes/stats/thread_pool,os,jvm

A persistently non-zero queue on the search pool means you are CPU-bound on search — the cluster is saturated, not slow. rejected climbing means you are past the queue limit and clients are getting errors, not just delay. Resist the urge to raise queue sizes; a bigger queue converts rejections into worse latency and nothing else.

Saturation has three honest responses: reduce work per query (steps 1-2), reduce queries (cache at the application layer, collapse duplicate dashboard refreshes), or add capacity. Do the first two first, because the third scales your bill linearly with the waste.

While you are in node stats, look at the JVM. Old-generation GC running constantly, or heap sitting above roughly 75% after collection, produces exactly the pattern people describe as "random slowness". Circuit breaker trips in the logs (parent or request breaker) are a related signal: the cluster is telling you a query class needs more heap than it should. Keep heap at or below 31 GB per node so compressed object pointers stay in play, and treat repeated breaker trips as a query-shape bug rather than a sizing accident.

Step 4: shard count and data volume per query

A search touches every shard of every index it matches, and each shard is a unit of work on a thread. Two opposite failure modes:

  • Too many small shards. A query across 600 tiny shards pays coordination overhead 600 times. Consolidate with rollover sizing, shrink, or fewer primaries per index.
  • Too few enormous shards. One 200 GB shard cannot be parallelized further, so its latency is your floor.

Time-series data has a free win here: query with a time range and make sure the request can prune. Data streams plus a bounded @timestamp filter let the coordinating node skip indices entirely — a dashboard asking for "last 15 minutes" should not be reading 90 days of shards, and if it is, check for a missing range filter or a wildcard index pattern that defeats pruning.

Step 5: caches, refresh, and merges

The cheap wins live here once the query shape is sane:

  • Filesystem cache is the one that matters. Elasticsearch and OpenSearch both rely on the OS page cache for index reads. If your data-to-RAM ratio has quietly moved from 10:1 to 60:1, every query is now reading disk. This is the legitimate case for more memory.
  • Request cache caches whole aggregation responses for size: 0 requests — but only when the query is bit-for-bit identical. A dashboard sending now-15m as a literal timestamp per refresh never hits it; rounding to now-15m/m makes it cacheable.
  • Refresh interval. The default 1 s refresh is a write-path cost that also churns segments. For log and analytics indices, index.refresh_interval: 30s buys real headroom. For a product catalog where an edit must appear immediately, it does not — that is the trade-off, and it belongs to the product, not the cluster.
  • Segment count. A force merge on indices that are no longer written (warm-phase, via ILM or ISM) reduces per-query segment work measurably. Never force merge an index still receiving writes.

Step 6: if there are vectors in the query

Hybrid and semantic retrieval add their own latency knobs, and they are easy to set carelessly:

  • num_candidates on kNN drives most of the cost. It is a recall/latency dial — measure recall at a few values instead of guessing high.
  • Vector search wants its data in memory. If HNSW graphs do not fit in the page cache, latency falls off a cliff; quantization (int8_hnsw and friends) is the standard answer and costs a little recall.
  • Filtered kNN behaves differently from filtered lexical search. A very restrictive filter can make approximate search work much harder, and exact search over the filtered set is sometimes faster. Test both.
  • Inference time counts. If embeddings are generated at query time, that call is part of your p99 whether or not it appears in took.

What we write down at the end

An engagement that starts as "search is slow" ends with a short table: the query shapes in the tail, took before and after per shape, the p99 of the whole endpoint, and which lever moved which number. That table is what makes the next regression a ten-minute investigation instead of another week of guessing.

It is worth saying plainly: sometimes the answer is more nodes, or faster disks, or more RAM per node. But that conclusion should come with a profile output attached. Capacity bought without one tends to get spent twice.

The short version

  1. Separate took from client-observed time before touching the cluster.
  2. Slow log to find the query shapes; remember thresholds are per shard.
  3. Profile the worst one; look for the dominant clause, agg cost, and shard skew.
  4. Check search thread pool queue and rejected — waiting is not the same as slow.
  5. Verify shard count and that time filters prune indices.
  6. Fix filesystem cache ratio, request-cache eligibility, refresh interval, segment count.
  7. For vectors, tune num_candidates and keep the graphs in memory.

If your p99 has drifted and nobody can point at the query shape responsible, that is the engagement we run: measure first, then change one thing at a time. Tell us what the numbers look like.