logsdb index mode: cutting log storage without losing the data

Most observability clusters we review are storage-bound, not CPU-bound. The hot tier is sized by how many days of logs have to stay on SSD, and the bill scales almost linearly with bytes on disk. ILM and data tiers move those bytes to cheaper storage. logsdb index mode attacks the other side of the equation: how many bytes each document takes in the first place.

This is a tutorial about what logsdb actually does, what you give up, and how to measure the result on your own data instead of trusting a vendor percentage.

What logsdb is

logsdb is an index mode in Elasticsearch — index.mode: logsdb, set on an index template, not on a live index. It is not a new engine. It is a bundle of three settings that were already available individually, applied together and tuned for log-shaped data:

1. Index sorting on host.name ascending and @timestamp descending by default. Sorting groups similar documents next to each other, which makes the columnar compression in Lucene far more effective on fields like host.name, service.name, log.level, and kubernetes.pod.name.
2. Synthetic _source. The original JSON document is not stored. _source is reconstructed on demand from doc_values and stored fields when someone actually asks for it.
3. Codec and field defaults suited to logs, including more aggressive compression than a stock index.

The saving comes mostly from item 2. In a typical ECS-shaped log document, the stored _source is a large share of the index footprint, and it is pure duplication of data already held in doc_values.

Availability moved around across releases: logsdb reached general availability in the 8.17 line, and recent Stack versions apply it by default to new logs-*-* data streams under some licenses. Do not assume — check GET my-index/_settings?filter_path=**.index.mode on a real index in your cluster before you plan around it.

Turning it on

Index mode is fixed at index creation, so you enable it in the template and pick it up at the next rollover:

PUT _index_template/logs-app-test
{
  "index_patterns": ["logs-app.test-*"],
  "data_stream": {},
  "priority": 500,
  "template": {
    "settings": {
      "index.mode": "logsdb",
      "index.number_of_shards": 1,
      "index.number_of_replicas": 1
    }
  }
}

POST logs-app.test/_rollover

Existing indices keep their old mode until they age out of retention. That is fine — it also means the change is gradual and reversible: revert the template, roll over again, and new indices go back to standard mode. No reindex, no downtime, no data loss. That property is why this is one of the safer cost levers available.

What you give up

Synthetic _source is a reconstruction, not a recording. The document you get back from a search is semantically equivalent, not byte-identical. Concretely:

• Array order is not preserved, and duplicates are collapsed. ["b","a","a"] comes back as ["a","b"]. If any downstream consumer depends on the order of a tags array, that is a behavior change.
• Field order and formatting are normalized. Whitespace, key order, and numeric formatting (1.0 vs 1) may differ.
• Fields must be reconstructable. A field indexed with doc_values: false and no stored value cannot be rebuilt. Objects mapped with enabled: false, and some specialized types, need synthetic_source_keep or a stored fallback.
• ignore_above and ignore_malformed values are retained separately, so they survive, but they land in _ignored_source rather than where you left them.
• Fetch cost rises. Rebuilding _source is per-hit work. A query returning 10 hits will not notice. A _search with size: 1000, a Discover session pulling wide documents, or a reindex reading millions of docs does more work per document.

There is also a modest ingest CPU cost from index sorting. On the clusters we have measured it is single-digit percent, but if your hot nodes are already pinned at 80% CPU during peak ingest, measure before you commit.

The two cases where we tell clients to hold off: pipelines that reindex log indices wholesale into another cluster on a schedule, and anything that treats the raw _source as an archival record of the exact bytes received. If you need the exact bytes, ship them to object storage separately; do not pay for the privilege on SSD in your hot tier.

Measuring it on your own data

Vendor numbers are averages over someone else's documents. Your saving depends on field count, cardinality, and how much of your document is text versus keywords. Run the A/B — it takes an afternoon.

Send the same stream to two data streams for 24 hours: logs-app.control on the default mode and logs-app.test with index.mode: logsdb. A tee output in Logstash or a second elasticsearch exporter in the OTel Collector is enough. Then compare bytes per document, not total bytes:

GET _cat/indices/logs-app.*?v&h=index,docs.count,pri.store.size&bytes=b

Divide pri.store.size by docs.count for each. On ECS-shaped container logs we typically see the logsdb side land between 40% and 60% of the control, with the wide, high-cardinality documents saving most. Narrow documents with one big unstructured message field save the least, because the message text dominates and it is stored either way.

Before you sign off, force merge both to one segment so you are not comparing different merge states:

POST logs-app.control/_forcemerge?max_num_segments=1
POST logs-app.test/_forcemerge?max_num_segments=1

Then check the read side. Replay your five most expensive saved Kibana queries against each and compare took, plus one deliberately ugly fetch (size: 500, no _source filtering) to see the reconstruction cost at its worst. Record the numbers. A 50% storage cut that adds 300 ms to every dashboard panel is a trade, not a win, and you want to be able to say which one you took.

Where it fits with ILM

logsdb and ILM are complementary, and they compound. ILM decides which tier the bytes live on; logsdb decides how many bytes there are. A hot tier holding 7 days at half the size is half the SSD, and the warm and cold copies downstream shrink with it. Snapshots shrink too.

One interaction worth knowing: the warm-phase shrink and forcemerge actions work normally on logsdb indices. Index sorting means force merge has more to work with, not less.

For metrics, the equivalent is index.mode: time_series (TSDS), which adds dimension-based routing and time-bounded indices on top of the same synthetic _source machinery. Same evaluation method: A/B, bytes per document, check the read path.

OpenSearch equivalents

OpenSearch has no logsdb index mode. The pieces are available separately and get you part of the way:

• index.codec: zstd or zstd_no_dict (2.9+) — a straightforward stored-fields compression win over the default, usually 10–20% on log data, with lower decompression cost than best_compression.
• Index sorting via index.sort.field / index.sort.order, configured at index creation. Sorting by host.name then @timestamp gives you the compression benefit of the logsdb sort without the synthetic _source part.
• Derived fields (2.15+) — compute fields at query time from _source instead of indexing them. Useful for rarely-queried fields on high-volume indices, and a different lever than logsdb but pointed at the same bill.
• Disabling _source is possible and we almost never recommend it: no reindex, no update-by-query, no document view in Discover.

If you are choosing between platforms and logging storage is the dominant cost line, this is a real difference in Elasticsearch's favor today. It is not on its own a reason to migrate — see our post on what actually differs between the two — but it belongs in the spreadsheet.

A short checklist

1. Confirm your version's logsdb status and licensing with GET _settings on a live index.
2. Stand up a control and a test data stream from the same source for 24 hours.
3. Force merge both, compare pri.store.size / docs.count.
4. Replay your five worst dashboard queries and one wide fetch against each; record took.
5. Check that nothing downstream depends on array order or exact _source bytes.
6. Flip the template for one noisy, low-risk data stream first. Roll over. Watch for a week.
7. Then the rest.

If your logging bill is the line item that started this conversation, storage per document is usually the second-biggest lever after retention itself — and unlike retention, it costs you no data. If you want a second pair of eyes on the measurement or on the ILM policy underneath it, get in touch.