Time series data streams and downsampling: cutting metrics storage without losing the dashboards

Logs got a lot of attention lately — logsdb index mode, tiering, retention policy. Metrics quietly kept growing. On most observability clusters we review, metrics are 20-40% of the storage bill and nearly 100% of the documents nobody ever reads at original resolution after the first week. Time series data streams (TSDS) plus downsampling fix that at the index level, without changing what your dashboards query.

This is how the two features work, what they cost you, and the order to roll them out.

What TSDS actually changes

A time series data stream is a data stream where the index is told, up front, that the documents are metrics: each document is a sample of a time series identified by a set of dimension fields, at a timestamp.

You declare it in the index template:

PUT _index_template/metrics-app
{
  "index_patterns": ["metrics-app-*"],
  "data_stream": {},
  "template": {
    "settings": {
      "index.mode": "time_series",
      "index.routing_path": ["host.name", "service.name", "metric.label"],
      "index.look_back_time": "2h"
    },
    "mappings": {
      "properties": {
        "@timestamp":   { "type": "date" },
        "host.name":    { "type": "keyword", "time_series_dimension": true },
        "service.name": { "type": "keyword", "time_series_dimension": true },
        "cpu.pct":      { "type": "double",  "time_series_metric": "gauge" },
        "requests.total": { "type": "long",  "time_series_metric": "counter" }
      }
    }
  }
}

Three mechanical consequences:

• Documents are sorted and routed by time series. Samples of the same series land in the same shard and adjacent in the segment, so the values of a given metric are stored next to each other. That is what makes compression work: a gauge that changes slowly compresses far better in series order than in arrival order.
• _id is synthesized from the dimensions plus the timestamp, which deduplicates re-sent samples for free. Agents that retry no longer double-count.
• _source is synthetic. The stored _source is reconstructed from doc values instead of kept verbatim, which is the bulk of the storage saving.

In our reviews, moving an existing metrics stream to index.mode: time_series lands somewhere around 40-60% smaller on disk for typical infra metrics with a handful of dimensions. Wide, high-cardinality dimension sets do worse. Measure on your own data; the number depends almost entirely on dimension cardinality and how noisy your gauges are.

OpenSearch does not have TSDS in this form. The comparable levers there are index sorting on the dimension fields with best_compression, and ISM rollup jobs for resolution reduction. The strategy in this post still applies; the syntax does not.

The constraints nobody reads until it bites

TSDS is a stricter index than you are used to. Before you convert anything:

• Writes are time-bounded. An index accepts documents only between index.time_series.start_time and end_time, derived from look_back_time (default 2h) and look_ahead_time (default 2h). Backfilling last month's metrics into a TSDS will be rejected. If you replay data, raise look_back_time on the template before you start, or backfill into a separate non-TSDS index.
• No updates, and no delete-by-ID workflow in the normal sense. Metrics are append-only; if you were doing anything else, this is a redesign.
• routing_path must match your dimensions, and dimensions must be keyword, ip, or a numeric type flagged with time_series_dimension: true. The dimension count is capped (16 by default via index.mapping.dimension_fields.limit).
• Synthetic _source reconstructs, it does not echo. Field order, array order for non-dimension arrays, and exact numeric formatting can differ from what you indexed. If something downstream does a byte comparison on _source, test it.

None of these are dealbreakers for metrics. All of them are dealbreakers if you try to put logs with unpredictable fields into a TSDS — that is what logsdb mode is for.

Downsampling: the second, bigger lever

TSDS shrinks each sample. Downsampling removes samples.

A downsample action rolls a source index into a new index at a coarser interval, replacing each metric with statistical aggregates over the bucket — min, max, sum, value_count, and the last value — per time series. A 10-second scrape interval reduced to 5 minutes keeps 1 document where there were 30.

Wire it into ILM:

"warm": {
  "min_age": "2d",
  "actions": {
    "downsample": { "fixed_interval": "5m" }
  }
},
"cold": {
  "min_age": "14d",
  "actions": {
    "downsample": { "fixed_interval": "1h" }
  }
},
"delete": { "min_age": "395d", "actions": { "delete": {} } }

Run the arithmetic for a realistic stream: 50,000 active series scraped every 10 seconds is 432M samples a day. Keep two days at full resolution, then 5-minute buckets for twelve days, then 1-hour buckets for the remaining thirteen months. Sample count for the long tail drops by a factor of 360 against raw resolution. Storage follows, roughly — downsampled documents carry more aggregate fields each, so call it two orders of magnitude, not three.

Downsampled indices stay in the same data stream and aggregations resolve across mixed-resolution backing indices. Dashboards over a 90-day window keep working with no query change.

What downsampling costs you

State it plainly, because someone will ask during an incident review:

• You lose the spike shape. A 300 ms latency excursion inside a 1-hour bucket survives as max, not as a curve. Alerting on high-resolution data past the downsample age becomes impossible.
• Averages need care. After downsampling, the correct average is sum / value_count, not the average of the bucket averages. Kibana handles this for TSDS metric fields; hand-written query DSL and external consumers may not.
• Percentiles are gone. min/max/sum/count/last value do not reconstruct a p95. If you need long-retention percentiles, keep a histogram field or accept the loss.
• Downsampling is one-way. Inside ILM the source index is removed once the downsample succeeds. Your snapshot is the only path back.

Choose the first downsample age by asking one question: how far back does anyone actually investigate at raw resolution? For most teams the honest answer is 48 hours, occasionally 7 days. Whatever they say, check it — your query logs and the time ranges people actually select in Kibana will tell you.

Rollout order that does not break dashboards

1. Measure the baseline. Per data stream: GET _cat/indices/metrics-*?bytes=gb&s=store.size:desc and the daily ingest volume. Without a before number the project has no result.
2. Count your dimensions. Pick the fields that identify a series and check cardinality: a pod.uid dimension on a churning Kubernetes cluster creates a new series on every restart and will blow up your series count. Prefer stable identifiers; keep ephemeral ones as non-dimension fields.
3. Convert forward, not backward. Update the index template to index.mode: time_series and roll over. New backing indices are TSDS; existing ones age out under the old policy. No reindex, no downtime.
4. Compare a full day. One day of TSDS backing index against one day of the old one, same stream, store.size per million documents. That ratio is your real compression number.
5. Add downsampling in cold first, at a coarse interval, where nothing is alerting. Watch the dashboards that use that range for a week.
6. Then move downsampling earlier — warm at 5 minutes — once you have confirmed nobody queries that window at raw resolution.
7. Re-measure and write the number down. "Metrics storage per day went from 420 GB to 96 GB" is the sentence that funds the next piece of work.

When not to bother

If metrics are under 10% of your cluster's storage, this is not your cost problem — look at logs and replica counts first. If your metrics already live in a purpose-built TSDB and only summaries land in Elasticsearch, TSDS buys you little. And if your team genuinely investigates at 10-second resolution across months, downsampling is the wrong tool; cheaper tiers are the right one.

Everywhere else the combination is unusually good value: a template change plus two ILM actions, no query rewrites, and a metrics line item that stops growing linearly with your fleet.

If you want the numbers run against your own cluster before you commit, that measurement is the first step of our cluster and cost review — tell us your daily metrics volume, scrape interval, and retention target and we can size the saving fairly quickly.