+1 (806) 515-4974
QueryCrunch

Cross-cluster search vs replication: one view over many clusters, without paying twice

Every cluster eventually outgrows being one cluster. A second region appears for latency or residency reasons, a security team wants audit logs isolated from application logs, or an acquisition arrives with its own Elasticsearch deployment and nobody wants to merge the two. Then someone asks the obvious question: can we still search all of it from one Kibana?

Yes. Cross-cluster search (CCS) and cross-cluster replication (CCR) are the two mechanisms, they solve different problems, and teams routinely reach for the expensive one when the cheap one would have done. This is how we decide, and what breaks in production.

The two mechanisms in one paragraph each

Cross-cluster search queries remote clusters in place. The local cluster fans a search out to the remotes over the transport protocol, each remote executes its share, and the coordinating node merges results. No data moves and nothing is stored twice. You pay in query latency — the slowest remote sets your floor — and in a hard dependency on the link.

Cross-cluster replication copies indices. A follower index on cluster B pulls operations from a leader index on cluster A and stays read-only, near-real-time behind it. Queries then run locally against a full copy. You pay in storage (every replicated byte exists twice, plus its replicas) and in a background job that can and will fall behind. CCR is an Elastic commercial feature; on OpenSearch the equivalent is the cross-cluster replication plugin, with the same leader/follower model and similar operational behavior.

Rule of thumb: CCS for reads across clusters, CCR for availability and locality. If what you actually need is "one dashboard over three regions", CCS is the answer and it costs nothing in storage.

Setting up a remote connection

PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "eu-logs": {
          "mode": "proxy",
          "proxy_address": "eu-logs.example.internal:9400",
          "server_name": "eu-logs.example.internal"
        }
      }
    }
  }
}

Two connection modes exist. sniff discovers gateway nodes and connects to them directly — fine inside one network, bad through a load balancer or NAT, because the addresses it learns may not be reachable from where you are. proxy mode sends everything through a single endpoint. On anything crossing a cloud boundary, start with proxy; it is the mode that survives the network team's next change.

Then query across them with the cluster:index syntax:

GET logs-*,eu-logs:logs-*,us-logs:logs-*/_search
{
  "query": { "range": { "@timestamp": { "gte": "now-1h" } } }
}

Kibana data views accept the same notation, which is usually all the "single pane of glass" anyone wanted.

What CCS actually costs at query time

Three things determine whether CCS feels fine or feels broken:

  • Round trips. A standard search is two phases — query, then fetch. Across a 90 ms link that is 180 ms of pure network before anything executes. ccs_minimize_roundtrips: true (the default for _search) pushes a full search to each remote and merges once, which is dramatically better on high-latency links. It is not compatible with scroll or with some async workflows, so check before assuming it is on.
  • The slowest remote. A fan-out is as fast as its worst participant. One overloaded region makes every global dashboard slow, and the symptom shows up on the local cluster where nothing is wrong.
  • Failure behavior. skip_unavailable: true on a remote means searches return partial results instead of an error when that cluster is down. That is the right setting for a convenience dashboard and the wrong setting for a compliance query. Decide per remote, deliberately, and make sure whoever reads the dashboard knows which one they are looking at.

Bandwidth is the quiet line item. Aggregations ship compact results, but a query that fetches 10,000 documents per region pulls those documents across an inter-region link every time someone refreshes a panel. Cross-region egress is billed. Keep global dashboards aggregation-shaped and set modest size values.

Where CCR earns its storage

Use CCR when you need one of these, and otherwise don't:

  1. Disaster recovery with a real RPO. Snapshots restore in hours; a follower index is seconds behind and promotes in minutes. If your recovery objective is measured in minutes, snapshots alone do not meet it.
  2. Read locality. Analysts in Singapore querying a US cluster feel every round trip. Replicate the indices they use and the queries run locally.
  3. Reference data everywhere. Small, slow-changing indices — product catalogs, entity lookups, enrichment sources — are cheap to replicate and remove a cross-cluster dependency from the hot path.

The pattern to avoid is replicating everything "for safety". We have seen a 40 TB log estate mirrored in full to a second region where nobody ever ran a query against the copy. That is a doubled storage bill plus doubled ingest-side write amplification, purchased as insurance against an event that snapshots already covered.

Operating CCR without surprises

  • Leader indices need soft deletes retained. Replication reads the operation history; index.soft_deletes.retention_lease.period (default 12 hours) is how long the leader keeps operations for a lagging follower. A follower disconnected longer than that cannot resume incrementally — it needs a full re-bootstrap from a remote recovery. Twelve hours sounds generous until a holiday-weekend network incident.
  • Followers are read-only. You cannot write to them, and you cannot reindex into them. Promotion is an explicit step: pause replication, close, unfollow, open. Rehearse it. A DR mechanism you have never promoted is a hypothesis, not a plan.
  • Mapping and setting changes propagate; ILM does not, automatically. Run a policy on the follower side that is aware it is following, or you will get a follower index deleted out from under an active leader — or worse, a leader rolled over while the follower's policy keeps the old index hot forever.
  • Watch the lag. GET _ccr/stats exposes operations_written, failed_read_requests, and the time since last read. Alert on follower lag directly, in seconds, with a threshold well under the retention lease period. Lag is the metric that tells you a promotion today would lose data.
  • Version skew has direction. A follower must be the same version as the leader or newer. That constrains upgrade order: upgrade followers first, then leaders. Plan the sequence before the maintenance window, not during it.

A layout that usually works

For a two-region observability estate: ingest locally in each region into local data streams, run ILM locally, connect both clusters as remotes to a single Kibana, and query with *:logs-*. No replication, no duplicate storage, one view. Then replicate only the handful of indices that genuinely need to be local or recoverable — alerting state, enrichment lookups, the last 7 days of the stream your on-call actually pages against — and leave the 350-day tail single-copy in frozen storage behind CCS.

That split is the whole decision: search remotely by default, replicate by exception, and write down the exception. Every byte you copy is a byte you pay for twice and a job you now have to monitor.

If you are drawing this out for a second region or a residency requirement, we are happy to look at the specific shape of it — the trade-offs change a lot depending on where the writes originate.