Backups you have actually restored: SLM, repository design, and restore rehearsals

Most clusters we review have snapshots. Far fewer have a restore anyone has timed. That gap matters because the two questions an incident asks are not "do you have a backup" but "how much data will we lose" and "how long until search is answering again". Those are RPO and RTO, and you cannot know either from a green snapshot list.

This is how we set snapshots up so both numbers are measured rather than assumed. Everything here applies to Elasticsearch with SLM and to OpenSearch with snapshot management (SM); the JSON differs, the design does not.

What a snapshot actually is

A snapshot is an incremental copy of the Lucene segment files backing each shard, written to a registered repository (usually S3, Azure Blob, GCS, or a shared filesystem). Three consequences follow, and they explain almost every surprise:

• Incremental at segment granularity. A new snapshot uploads only segments not already in the repository. A force merge rewrites segments, so the snapshot right after a merge is nearly full size again. If your warm phase force merges nightly, your "incremental" nightly snapshot is not.
• Repository-scoped, not index-scoped. Deleting one snapshot only frees segments no remaining snapshot references. Storage does not drop the way people expect when they prune one day.
• Point-in-time per shard, not per cluster. Data indexed during the snapshot may or may not be included. For time-series data that is fine; for a mutable catalog index, assume your RPO is the snapshot start time.

Repository design

Keep it boring:

• One repository per cluster per purpose. Do not share a repository between two clusters that both write to it. Two writers corrupt repository metadata; this is the single most common way teams lose a backup history.
• Same region as the cluster for the operational repository. Cross-region copies are a separate, slower repository, not your primary.
• Set max_snapshot_bytes_per_sec deliberately. The default recovery/snapshot throttles exist to protect the cluster, and they also set your restore floor. If your repository can do 500 MB/s and the node-level restore rate is throttled to 40 MB/s, your RTO is defined by the throttle, not the network.
• Lifecycle rules on the bucket are a trap. An S3 lifecycle policy that transitions objects to an archive class, or expires them, will break snapshots that still reference those segments. Exclude the repository prefix, or manage retention only through SLM/SM.

Registering a repository, minimal version:

PUT _snapshot/prod-backups
{
  "type": "s3",
  "settings": {
    "bucket": "acme-es-prod-backups",
    "base_path": "cluster-a",
    "max_snapshot_bytes_per_sec": "200mb",
    "max_restore_bytes_per_sec": "200mb"
  }
}

Then verify it before trusting it:

POST _snapshot/prod-backups/_verify
POST _snapshot/prod-backups/_analyze?blob_count=100&concurrency=10

The analyze API writes and reads test blobs and fails loudly on object stores that are not consistent enough to be a repository. Run it once per new repository; it takes minutes and has caught misconfigured S3-compatible storage for us more than once.

SLM policies that match the data

One policy for everything is the usual starting point and the usual reason snapshots collide with peak ingest. Split by how much loss each stream can tolerate.

PUT _slm/policy/hourly-critical
{
  "schedule": "0 15 * * * ?",
  "name": "<critical-{now/H{yyyy.MM.dd.HH}}>",
  "repository": "prod-backups",
  "config": {
    "indices": ["orders*", "catalog*", ".kibana*"],
    "include_global_state": true
  },
  "retention": { "expire_after": "30d", "min_count": 24, "max_count": 400 }
}

Notes that matter in practice:

• include_global_state: true on at least one policy. It carries index templates, ILM policies, ingest pipelines, and cluster settings. A restore without them gives you data with no lifecycle management and no pipelines — technically recovered, operationally broken. Keep it false on the high-frequency log policy to save churn.
• Schedule off-peak, and stagger. Snapshot I/O competes with indexing. Two policies firing at 0 0 * * * ? guarantee contention.
• Retention needs min_count. Time-based expiry alone can leave you with zero snapshots if the policy stops running and then resumes.
• Log streams often need only daily snapshots, because ILM plus replicas already cover node loss and the cost of losing six hours of debug logs is not the cost of losing six hours of orders.

Check the policy is actually succeeding, not just existing:

GET _slm/stats
GET _slm/policy/hourly-critical?human

snapshots_failed climbing while the last success sits two weeks back is the state we find most often. Alert on it — SLM failures are a metric you can route to Kibana alerting in ten minutes, and it is the highest-value alert in this whole article.

The restore rehearsal

This is the part that turns a hypothesis into a number. Do it quarterly, and after every major version upgrade.

1. Pick the scenario. "We lost the primary search index" is a different rehearsal from "we lost the cluster". Start with the index-level one; it is cheap and it is the one that actually happens.
2. Restore to a different name so nothing in production is at risk:

POST _snapshot/prod-backups/critical-2026.06.08.15/_restore
{
  "indices": "catalog-000042",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1",
  "include_aliases": false,
  "index_settings": { "index.number_of_replicas": 0 }
}

include_aliases: false keeps the restored copy out of the live alias. Dropping replicas to zero makes the restore finish faster; add them back after.

3. Time it and watch throughput. GET _recovery?active_only=true shows bytes and percent per shard. Compute GB/minute. That number, times your largest index, is your real RTO — and it is usually two to five times what people guess.
4. Verify content, not just green status. Compare _count, spot-check a few document IDs, and run one real query. A restore can complete with an index that has the wrong mappings because the template it depended on was not in the snapshot.
5. Write down the numbers — snapshot age at restore time (your RPO), minutes to green (your RTO), throughput, and anything you had to look up mid-restore. That last list becomes the runbook.
6. Clean up. Delete the restored-* indices; they are counting against disk and shard limits.

If the measured RTO is worse than the business expects, the fixes are known and ordered: raise the restore throttle, restore with zero replicas and add them later, restore only the indices the application needs first, or keep a warm standby via cross-cluster replication (Elastic CCR, or cross-cluster replication on OpenSearch) instead of relying on restore at all.

Version and platform compatibility

The rules here are strict and worth memorizing before you need them:

• A snapshot can be restored into the same major version, or the next one — 7.x snapshots restore into 8.x, not into 9.x. This is why a snapshot taken years ago is not the archive you think it is, and why "restore the old snapshot" is not an upgrade rollback plan two majors later. Rollback plans need a snapshot taken immediately before the upgrade.
• Elasticsearch and OpenSearch snapshots are not freely interchangeable. OpenSearch can read Elasticsearch 7.x-era snapshots; snapshots from later Elasticsearch versions are not a supported migration path, and the reverse direction is not supported at all. For platform moves, plan on reindex-from-remote or a dual-write cutover — we covered the mechanics in the Elasticsearch-to-OpenSearch comparison.
• Frozen-tier indices are backed by searchable snapshots. Do not delete the snapshot that a frozen index is mounted from, and do not let a bucket lifecycle rule archive it. Losing that object does not degrade the index; it removes it.

A checklist you can run this week

• Repository registered, _verify and _analyze both clean.
• Bucket lifecycle rules exclude the repository prefix.
• At least two SLM/SM policies: frequent for stateful data with include_global_state: true, daily for logs.
• Retention set with both expire_after and min_count.
• An alert on SLM failures and on last-successful-snapshot age.
• One timed restore rehearsal in the last 90 days, with the RPO and RTO numbers written down.

It depends how far you take it. A 20-node logging cluster with 7-day retention may rationally accept a 6-hour RTO and skip standby entirely; a revenue-path product search index usually cannot, and the honest answer there is replication plus a rehearsed restore, not one or the other. Either way, pick the numbers on purpose — and measure them once before an incident measures them for you.

If you want a second pair of eyes on snapshot design, restore timing, or a DR plan you have not been able to rehearse, tell us about the cluster.