QueryCrunch | Hot spotting: why one data node is at 95% CPU while the rest idle

A cluster with six data nodes and one of them pinned at 95% CPU is not a capacity problem, even though it will be reported as one. Indexing rejections, climbing bulk queue depth, and a p99 that tracks a single node's GC pauses all look like "we need bigger nodes" from the dashboard. They usually mean work is unevenly distributed, and adding a seventh node spreads the idle capacity further while the hot shard stays exactly where it was.

This is the triage sequence we run, and the settings that move load once you know what is causing the imbalance.

Step 1: confirm it is a hot spot, not a busy cluster

Two commands, in this order.

GET _cat/nodes?v&h=name,node.role,cpu,load_1m,heap.percent,disk.used_percent,ram.percent
GET _cat/thread_pool/write,search?v&h=node_name,name,active,queue,rejected,completed

What you are looking for is spread, not level. If every node sits at 70% CPU, you have a capacity problem and the rest of this post does not apply. If one node is at 95% and five are at 25%, and the write pool shows rejected incrementing on that one node only, you have a hot spot.

Then find out what is on it:

GET _cat/shards?v&h=index,shard,prirep,state,docs,store,node&s=node,store:desc
GET _nodes/hot_threads?threads=3

hot_threads tells you which kind of work is hot — bulk indexing, merge, search, or GC. That single distinction decides which of the following four causes you are dealing with.

Cause 1: the write-active shards landed on the same node

The classic. A time-based data stream rolls over nightly; the allocator places the new write index's primaries by shard count, which is blind to the fact that in a logging cluster roughly all of the write traffic goes to the newest index. Two of three write-active primaries on one node means that node takes two-thirds of the ingest.

Modern Elasticsearch (8.6+) replaced the old greedy allocator with the desired balance allocator, which weighs shard count, disk usage, and — from 8.8 — write load. The relevant cluster settings:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.balance.shard": 0.45,
    "cluster.routing.allocation.balance.index": 0.55,
    "cluster.routing.allocation.balance.disk_usage": "2e-11",
    "cluster.routing.allocation.balance.write_load": 10.0
  }
}

Raising balance.write_load makes the allocator treat ingest-heavy shards as heavier objects and pull them apart. Check the allocator's own view before and after:

GET _internal/desired_balance

That endpoint shows the balance the cluster is trying to reach and how far it has to move to get there. If desired and current already agree and the node is still hot, the allocator is not your problem — keep reading.

On OpenSearch, and on Elasticsearch versions before the desired-balance allocator, the blunt instrument still works:

PUT logs-app-000042/_settings
{ "index.routing.allocation.total_shards_per_node": 1 }

Apply it in the index template so every rollover inherits it. One caveat worth stating plainly: set it too tight and shards become unassignable during a node restart, because there is nowhere legal to put them. With N data nodes and an index of p primaries and r replicas, keep total_shards_per_node at least ceil(p * (1 + r) / (N - 1)) so you can lose a node and still allocate.

Cause 2: custom routing concentrated a tenant

If you use _routing — per-customer, per-device, per-tenant — documents hash to a single shard by design. That is the point: one-shard queries are fast. It also means your largest tenant is one shard, on one node, and no allocator setting will split it.

Confirm it by comparing per-shard doc counts on the index:

GET _cat/shards/orders?v&h=shard,prirep,docs,store,node&s=docs:desc

A 40x spread between the largest and median shard is a routing skew, not an allocation bug. The fixes, in ascending order of effort:

Routing partitions. index.routing_partition_size: 8 spreads one routing value across 8 shards instead of 1. Queries must still supply the routing value, and the index must not have parent-join fields. This is the cheap fix and it works.
Split the whale out. Give tenants above a size threshold their own index behind the shared alias. Query the alias; the big tenant gets its own shard budget.
Drop custom routing. Sometimes the single-shard query win was never measured and the skew is pure cost. Measure before you defend it.

Cause 3: one shard is simply too big

A 400 GB shard next to a set of 40 GB shards hot-spots on merges and on any query that touches it, and it cannot be rebalanced away — moving it just moves the problem, at the cost of 400 GB of network transfer.

This is a shard-sizing failure upstream of the allocator; the target is 20-50 GB per primary for search and log workloads, enforced with max_primary_shard_size on rollover. For an existing oversized index, the repair is _split (requires the index to be read-only and the target shard count to be a multiple of the source), or a reindex into a template with sane rollover conditions. Neither is instant — plan the I/O budget.

Cause 4: mixed roles on a node that should be dedicated

Check the node.role column from step 1. A node carrying m (master-eligible) alongside d will have its master duties starved during an ingest burst, and cluster-state updates back up behind bulk work — which presents as cluster-wide latency, not a single hot node. The same applies to an ingest-pipeline-heavy node running grok or GeoIP enrichment while also serving searches: hot_threads will show ingest frames and the CPU is real work that does not belong there.

Dedicated master nodes on any cluster above three data nodes, and coordinating-only nodes in front of heavy aggregation traffic, are boring and cheap. If ingest processing is the hot work, move the pipelines to dedicated ingest nodes or push the parsing upstream to Logstash or the OTel collector, where it scales horizontally and independently of your data tier.

What not to do first

Do not add nodes. Until the allocator's placement logic is fixed, new nodes take an even share of shards and an uneven share of load. You pay more and the hot node stays hot.
Do not manually _cluster/reroute in a loop. It works for exactly as long as it takes the allocator to undo it, and manual moves accumulate into a state nobody can reason about at 3 a.m.
Do not raise thread pool sizes to absorb rejections. Write rejections are backpressure doing its job. Larger queues turn a fast failure your client can retry into heap pressure and a slow one it cannot.

Make imbalance visible before it pages you

Add two panels to the cluster health dashboard and an alert on each:

CPU spread — max node CPU minus median node CPU across data nodes. Alert above 30 points sustained for 15 minutes. This catches hot spotting while it is still a performance annoyance.
Write rejections by node — rejected from the write thread pool, differentiated, grouped by node. Any non-zero value on a single node with the rest at zero is a placement problem, not a load problem.

Both are available from Elastic Agent's or Metricbeat's elasticsearch module, and the OpenSearch equivalents through the node stats API. The measurement is worth more than any of the settings above, because it tells you which of the four causes you have before someone requisitions hardware.

The honest trade-off

Balancing by write load is not free: the allocator moves shards to achieve it, and shard relocation consumes network and disk on both ends. On a cluster that rolls over hourly, aggressive balance.write_load can keep shards in near-constant motion. Start at the defaults, raise the weight one step, watch _internal/desired_balance and relocation counts for a day, and stop when the CPU spread is under control. "It depends" here means: it depends on how often your write-active shard set changes.

If your cluster has one node everyone knows to worry about, we do this as a fixed-scope piece of a cluster and cost review — measurement first, then the smallest change that flattens the curve. Tell us what you are seeing and include the _cat/nodes output.