Bulk rejections and 429s: tuning the write path before you add data nodes

Every few months a client sends the same screenshot: a Logstash or Elastic Agent log full of es_rejected_execution_exception, a bulk queue at 1000, and a question that is really a budget question — "how many more data nodes do we need?"

Usually the answer is none. Bulk rejections are a backpressure signal from one specific threadpool, and on most clusters we look at, the write path is being asked to do several times more work per document than it needs to. This post is the order we work through it: read the signal, fix the client, fix the index, and only then size the hardware.

What a 429 actually means

Elasticsearch and OpenSearch handle bulk indexing on the write threadpool. Its size is fixed at the number of allocated processors, with a bounded queue (200 by default in Elasticsearch 8.x and 9.x; OpenSearch 2.x/3.x uses 10000 for write but adds shard-level indexing backpressure on top). When every thread is busy and the queue is full, the node rejects the request with HTTP 429 and es_rejected_execution_exception.

That is the system working correctly. A rejection is cheap; an unbounded queue that eats heap and then OOMs the node is not. The bug is almost never "the queue is too small" — resist the urge to raise it.

Read the actual numbers before theorising:

GET _cat/thread_pool/write?v&h=node_name,active,queue,rejected,completed
GET _nodes/stats/indices,thread_pool,jvm?filter_path=nodes.*.name,nodes.*.indices.indexing,nodes.*.thread_pool.write,nodes.*.jvm.mem.heap_used_percent

Two patterns, two different problems:

• Rejections spread evenly across all data nodes. The cluster genuinely cannot absorb the write rate at the current cost-per-document. Keep reading.
• Rejections concentrated on one or two nodes. This is not a throughput problem, it is a balance problem — a hot shard, an oversized index with too few primaries, or a node also carrying an expensive search workload. Fix the distribution first; adding nodes to an unbalanced cluster adds idle nodes.

Also check indices.indexing.index_time_in_millis / index_total per node. That is your average cost per document. If it is above about 1 ms for ordinary log documents, the problem is in the index, not the client.

Fix the client first — it is usually free

Three client-side settings explain most of what we find.

Bulk request size. Aim for requests of 5–15 MB of body, not a fixed document count. A 500-document bulk of 2 KB logs is 1 MB — too small, and you pay per-request coordination overhead on every one. A 500-document bulk of 200 KB documents is 100 MB and will push a coordinating node's heap around. Size by bytes. Logstash does this with pipeline.batch.size tuned against your average event size; most language clients have a byte-size flush threshold in their bulk helper.

Concurrency. The right number of in-flight bulk requests is roughly number_of_data_nodes × write_threadpool_size, and in practice two to three times the data node count is enough to saturate the path. More than that just fills queues and generates the 429s you are debugging.

Retry behaviour. Retry 429s with exponential backoff and jitter, and retry only the rejected items — a bulk response is per-item, and blindly replaying the whole request double-indexes everything that succeeded. The official clients' bulk helpers do this; hand-rolled HTTP loops often do not. If you are on a data stream with TSDS enabled, synthesised _id makes duplicate samples idempotent, which is one quiet reason to prefer it for metrics.

And one anti-pattern: do not send bulk requests with explicit _id values unless you need upsert semantics. An auto-generated ID lets the engine skip the "does this document already exist" lookup. On a log stream with billions of documents, that lookup is a meaningful share of index time.

Then fix what each document costs

Every document pays for every field you index. Most ingest pipelines index far more than anyone queries.

• refresh_interval. The default 1 s means a new segment roughly every second, and a merge treadmill behind it. For logs and metrics nobody queries sub-minute, set index.refresh_interval: 30s (or 60s). On a busy ingest stream this alone commonly cuts index time per document by 20–30% and takes real pressure off merging. The trade-off is explicit: data is not searchable until the next refresh. Keep 1 s only where a user types something and expects to see it immediately.
• Replicas during a backfill. number_of_replicas: 0 for a bulk reindex or historical load, then set it back and let recovery copy the finished segments. You halve the write work during the load. Only do this when a snapshot or the source system is your fallback — for live production ingest, keep replicas on.
• Fields you never query. index: false on fields you only display, doc_values: false on fields you never aggregate or sort, and norms: false on keyword-ish text. Disable the _source-adjacent extras you do not use. Also check for the classic waste: a text field with a .keyword multi-field that nothing ever aggregates on, applied to every string by the default dynamic mapping.
• Dynamic mapping. If new fields are appearing at ingest time, every one of them triggers a cluster-state mapping update, serialised through the master. A noisy JSON payload can turn your master node into the bottleneck. Pin a dynamic template, or set dynamic: false and add fields deliberately. (We wrote about the failure mode in more detail in mapping explosion.)
• Ingest pipeline cost. grok with a poorly anchored pattern, or several script processors per document, can dominate index time. Measure, do not guess:

GET _nodes/stats/ingest?filter_path=nodes.*.ingest.pipelines

That gives per-pipeline and per-processor time_in_millis and count. Divide. Any processor over ~0.2 ms per document is worth a look; a grok that is doing the work a dissect could do is the usual culprit.

• Index mode. For logs, index.mode: logsdb and for metrics time_series change the storage and the write profile. They cost a little more CPU at index time and save a lot of disk and merge I/O. On I/O-bound nodes that is a net win; on CPU-saturated nodes it is not. Measure which one you are.

Shards and merging

Writes go to primaries, so the number of primaries sets your write parallelism ceiling.

A single-primary index on a six-node cluster uses one node's write threads no matter how many bulk requests you send. The fix is rollover sized so that each primary takes 30–50 GB and the primary count is a sensible multiple of your data node count. More primaries than that and you pay for it in search fan-out and cluster state — the usual trade-off, covered in shard sizing.

Then check whether merging is the real limit:

GET _cat/nodes?v&h=name,disk.used_percent,heap.percent,merges.current,merges.total_time
GET _nodes/stats/indices/merges

If merges.current sits above zero continuously and your disks are near their IOPS limit, the cluster is throttling itself. The default indices.store.throttle behaviour assumes SSDs; on gp3 volumes with provisioned IOPS left at the floor, merging is the bottleneck and no client tuning will move it. Raise IOPS, or raise refresh_interval so there is less to merge. Do not raise index.merge.scheduler.max_thread_count on spinning or throttled storage — you make it worse.

Only then, add capacity

Having done the above, if the write path is still rejecting and you have confirmed:

• bulk bodies in the 5–15 MB range, concurrency near nodes × write threads,
• refresh_interval relaxed, pipeline processor time under control,
• primaries spread evenly, no single hot node,
• disk IOPS not throttled,

then you have an honest capacity problem, and now the sizing question has a defensible answer. Use measured index time per document and target write rate to size it, rather than doubling the cluster and hoping.

Two scaling notes worth knowing before you buy hardware. First, dedicated ingest or coordinating nodes help when ingest pipelines are the expensive part, because they move that CPU off the data nodes — they do nothing for a merge-bound cluster. Second, if your write volume is spiky rather than high, a buffer in front (Kafka, or Logstash with persistent queues) converts rejections into a queue you control, which is almost always cheaper than sizing for peak.

The one-hour version

If you only have an afternoon:

1. _cat/thread_pool/write?v — are rejections even or concentrated?
2. _nodes/stats/indices/indexing — index time per document.
3. _nodes/stats/ingest — per-processor cost; kill the worst grok.
4. Set refresh_interval: 30s on the high-volume write indices.
5. Fix bulk sizing to bytes and cap concurrency at 2–3× data nodes.
6. Re-measure the same three endpoints and write both numbers down.

We have seen that sequence take a cluster from steady rejections to 40% write-path headroom with no hardware change. We have also seen it confirm, in writing, that a cluster really did need more nodes — which is a perfectly good outcome, just a more expensive one.

If you want the measurement done against your own cluster before you sign off a bigger instance plan, that is the first step of our cluster and cost review. Send us your daily ingest volume, average document size, and current node layout and we can usually tell quickly which of the four causes above you have.