Minor upgrades are boring, and that is why teams assume major ones will be too. They are not. A major version removes settings, drops index formats, changes defaults, and breaks plugins and clients that nobody on the team owns. The upgrade itself is usually an hour of rolling restarts; the failures come from the three weeks of homework that got skipped.
This is the sequence we run on client clusters, Elasticsearch or OpenSearch. It is deliberately boring, and the expensive step is the rehearsal, not the cutover.
Rule zero: one major at a time, and read the version matrix
You can only jump one major version in place. Going from N-2 to N means N-2 → N-1 → N, or a reindex/remote-reindex into a fresh cluster. Before anything else, write down the actual version numbers for every moving part:
- Cluster nodes — target patch version, not just the major.
- Kibana / OpenSearch Dashboards — must match the cluster major, and upgrades after the cluster.
- Ingest components — Logstash, Beats, Elastic Agent, the OTel collector's exporters.
- Client libraries — the language clients your apps use, plus anything embedded in a framework.
- Plugins — analysis plugins (ICU, kuromoji, phonetic), repository plugins, and anything custom. Plugins are built per exact version. A plugin with no build for your target is a hard stop, and it is the single most common reason an upgrade gets postponed on the day.
If a component has no compatible version, the upgrade plan changes shape now, not during the change window.
Step 1: Sweep deprecations
Turn on deprecation logging and let it collect a full traffic cycle — a week, including whatever batch jobs run on Sundays. Deprecation warnings are emitted by real queries from real clients, so a quiet afternoon sample tells you nothing about the nightly report that still sends a removed parameter.
Check, at minimum:
- Deprecated or removed cluster and index settings, including ones baked into index templates you copied from a blog post in 2019.
- Old index formats. Indices created several majors ago often cannot be read by the new version even if they were carried forward by an earlier upgrade. They must be reindexed or deleted before you go.
- Query DSL and API changes — parameters that now warn, endpoints that moved, response shapes your dashboards parse.
- Mapping and template syntax that changed across the boundary.
Elastic ships an upgrade assistant in Kibana that surfaces much of this; treat it as the starting list, not the whole list, and pair it with the raw deprecation log.
Step 2: Snapshot, and prove the snapshot restores
PUT _snapshot/backups/pre-upgrade-2026-06-01?wait_for_completion=false
{ "indices": "*", "include_global_state": true }
A snapshot you have never restored is a belief, not a backup. Restore it somewhere — that restore is also step 3's input, so you get the verification for free.
One constraint that surprises people: snapshots are forward-compatible, not backward. A snapshot taken on the new version cannot be restored into the old one. That asymmetry is the whole reason the rollback story is what it is (see the end).
Step 3: Rehearse on a restored clone
Stand up a cluster at the current version, restore the snapshot into it, point a copy of real traffic at it, and perform the entire upgrade there. Scaled-down node counts are fine; the shapes you are testing are functional, not capacity.
What the rehearsal is actually for:
- Confirm every plugin installs and loads at the target version.
- Run the top queries and every saved dashboard, and diff the results against the old cluster. Not just "no error" — compare hit counts and top-N ordering. Scoring and default changes across majors can quietly reorder results, and a search team finds out from customers.
- Run the ingest path end to end: agent or collector → pipeline → index, with the real templates.
- Time it. Upgrading one node tells you how long thirty will take, and whether shard recovery saturates your network.
- Write down every manual fix you had to make. That list is the change ticket for production.
The rehearsal is the step teams cut for schedule reasons. It is also the only step that converts unknown risk into a checklist.
Step 4: The rolling upgrade
On the day, in order:
- Stop non-essential indexing and, if you can, flush: fewer translog operations to replay means much faster recovery per node.
- Disable replica allocation so the cluster does not start rebuilding shards for a node that will be back in four minutes:
PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.enable": "primaries" } }
- Upgrade node by node. Stop the node, install the new version, keep the data path, start it. Upgrade data nodes before master-eligible ones is a common convention; what matters more is that you never lose master quorum and you do one node at a time.
- Re-enable allocation and wait for green before touching the next node:
PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.enable": null } }
- Upgrade Kibana or Dashboards last, then the ingest components, then the clients.
A mixed-version cluster works during the rolling window, but it is a transition state, not a resting state. Do not start the upgrade on a Friday and finish on Monday: while versions are mixed, shards cannot relocate from newer nodes back to older ones, and your degraded-state tolerance shrinks with every hour.
Step 5: Verify with numbers you wrote down beforehand
Before the window, capture baselines: p95 search latency per key endpoint, indexing throughput, heap usage, hit counts for a set of canary queries. After the upgrade, take the same measurements and compare. "Feels fine" is not a verification step, and the regressions that matter — a slightly slower query, a changed default that doubles merge I/O — are invisible without the before number.
Also confirm the boring things: ILM or ISM policies still advancing, snapshot schedule still firing, alerting rules still evaluating, and every dashboard still rendering.
About the rollback
Be honest with stakeholders here. Once a node has started on the new version, its on-disk data is upgraded and that node cannot go back. The real rollback is: restore the pre-upgrade snapshot into a cluster at the old version, and accept the loss of everything ingested since the snapshot — which is why you also keep the source data replayable (a queue with retention, or ingest that can be re-run) for the duration of the window.
For clusters where that is unacceptable, the plan is different by design: build a parallel cluster at the target version, reindex from remote or dual-write into it, cut traffic over behind a proxy or alias, and keep the old cluster running until you are confident. It costs more and takes longer, and for a revenue-path search cluster it is usually the right trade.
The short version
One major at a time. Sweep deprecations from a full week of real traffic. Snapshot and restore it. Rehearse the entire upgrade on the clone and let it generate your checklist. Roll node by node with allocation disabled, Kibana last. Verify against baselines you captured first. And decide which rollback you actually have before you need it.
If you are staring at an upgrade with old indices, unowned plugins, and no rehearsal environment, that is the normal starting point — and the work is mostly discovery. Tell us about the cluster and we will tell you which of the two plans above it needs.