QueryCrunch | Mapping explosion: field limits, dynamic templates, and when to reach for flattened

Every few months we get the same email. A logging cluster is unstable — master node under pressure, long GC pauses, index requests rejected — and nobody changed the hardware or the ingest volume. Then we look at the mapping and one index template has 3,800 fields, most of them named things like kubernetes.annotations.checksum/config or payload.user_42.preferences.

This is field explosion. It is caused by dynamic mapping doing exactly what you told it to do, and it is one of the few cluster problems where the fix is a mapping change rather than more nodes.

What dynamic mapping actually does

By default, Elasticsearch and OpenSearch add a field to the mapping for every new JSON key they see. A string gets a text field plus a .keyword sub-field. That is convenient for the first week of a project and dangerous for any stream whose shape you do not control: Kubernetes annotations, third-party webhooks, customer-supplied metadata, stack traces serialized as objects, or anything with an ID in the key name.

The result is one mapping entry per distinct key ever observed, forever, because mappings are additive. You cannot remove a field from a mapping; you can only stop adding new ones and reindex into something cleaner.

What it costs

Three separate costs, and teams usually only notice the third:

Cluster state size. Mappings live in cluster state, which the master publishes to every node on every change. A few thousand fields across a few hundred indices turns into tens of megabytes of state being diffed and shipped. Symptoms: slow master, pending_tasks backing up, long index-creation times, rollover taking minutes.
Heap per shard. Field metadata is per-shard overhead. Wide mappings multiplied by many shards is a reliable way to spend heap on nothing anyone queries.
The hard stop. index.mapping.total_fields.limit defaults to 1,000. When a stream crosses it, indexing fails with Limit of total fields [1000] has been exceeded. The usual response is to raise the limit to 2,000, then 5,000, which converts a loud failure into the two quiet failures above.

Raising the limit is a valid emergency action to stop data loss. It is not a fix. Note the fields you added and come back.

Diagnose it first

Count fields before you theorize:

GET my-index/_mapping?filter_path=**.properties
GET my-index/_field_caps?fields=*

_field_caps is the practical one — it gives you every queryable field and its type. Pipe it through jq and count:

curl -s "$ES/my-index/_field_caps?fields=*" | jq '.fields | keys | length'
curl -s "$ES/my-index/_field_caps?fields=*" | jq -r '.fields | keys[]' \
  | cut -d. -f1-2 | sort | uniq -c | sort -rn | head -20

That last line is the money shot: it tells you which object prefix owns the explosion. Nine times out of ten it is a single subtree — annotations, labels, headers, or a params blob.

Then check whether anyone uses it. Kibana's field-usage stats or _stats?fields=* on request counts will tell you that a field appearing in 40 million documents has never been in a query. Those are free to stop indexing.

Fix 1: dynamic templates, so new fields land as something cheap

If you want the values searchable but not the full text + keyword treatment, map strings to keyword only and cap their length:

"dynamic_templates": [
  {
    "strings_as_keyword": {
      "match_mapping_type": "string",
      "mapping": { "type": "keyword", "ignore_above": 256 }
    }
  }
]

For logs this is almost always the right default. Full-text analysis on a container.id buys nothing and costs an analyzed field. ignore_above also protects you from a 2 MB stack trace becoming a single keyword term.

Fix 2: `dynamic: false` or `dynamic: strict` on the subtree you do not control

dynamic: false stores the JSON in _source but does not map it: the data is returned in search hits, it is just not queryable or aggregatable. dynamic: strict rejects the document instead.

"properties": {
  "kubernetes": {
    "properties": {
      "namespace": { "type": "keyword" },
      "pod":       { "type": "keyword" },
      "annotations": { "type": "object", "dynamic": false }
    }
  }
}

Use false for diagnostic payloads a human reads after they have already found the document by other fields. Use strict for a core schema where a surprise field means an upstream bug you want to hear about — but only where you have a dead-letter path, because strict mapping turns a schema change into dropped documents.

Fix 3: `flattened` when you need to query it but not model it

The flattened type maps an entire object subtree as one field. All leaf values become keywords under the parent name:

"labels": { "type": "flattened" }

Query the whole subtree or a specific path:

{ "term": { "labels": "production" } }
{ "term": { "labels.env": "production" } }

One mapping entry, unbounded keys, still searchable. The trade-offs are real and worth stating plainly: everything is a keyword (no numeric ranges, no dates, no full-text), no analysis, and it is awkward in Kibana's field picker because sub-paths are not declared. For arbitrary label and tag bags, that is a good deal. For a field you build dashboards on, map it explicitly.

OpenSearch has flat_object, which is close but not identical — notably weaker on some query types and on doc-value behavior. Test your actual queries before assuming parity.

Fix 4: `subobjects: false` and the OTel-shaped problem

OpenTelemetry attribute names are dotted: http.request.method, db.system. Classic mapping turns those into nested objects, which explodes badly and breaks when one document sends http.request as a string and another as an object — the mapping conflict that produces object mapping for [http.request] tried to parse field as object.

subobjects: false keeps dotted names as literal flat field names:

PUT otel-logs
{ "mappings": { "subobjects": false,
  "properties": { "http.request.method": { "type": "keyword" } } } }

If you are ingesting OTel data, set this from day one. Retrofitting it requires a reindex, because you cannot change subobjects on an existing mapping.

Fix 5: stop shipping the fields

The cheapest field is the one that never arrives. An ingest pipeline that drops the subtree is often better than any mapping trick:

{ "remove": { "field": "kubernetes.annotations", "ignore_missing": true } }

Or do it at the edge, in the Elastic Agent / Filebeat processor or the Logstash filter, and save the network and ingest CPU as well. Ask the question directly: has anyone queried this in the last 90 days? If not, drop it and keep the decision in version control next to the pipeline.

Guardrails worth setting now

"settings": {
  "index.mapping.total_fields.limit": 1000,
  "index.mapping.depth.limit": 10,
  "index.mapping.nested_fields.limit": 50,
  "index.mapping.total_fields.ignore_dynamic_beyond_limit": true
}

That last setting is the important recent addition: instead of rejecting the document when the limit is hit, it indexes the document and ignores the extra dynamic fields, recording them in _ignored. You keep the data and get a signal instead of an outage. Alert on _ignored being non-empty — it is the early warning that a stream's shape changed.

Cleaning up an index that is already wide

Mappings are additive, so cleanup means reindex:

Write the target template with explicit properties, a keyword dynamic template, and dynamic: false or flattened on the wild subtrees.
Reindex one recent index into it and compare store size, field count, and a real dashboard load. We commonly see 30-60% store reduction on log indices that were mapping every annotation, mostly from dropped text fields and doc values nobody read.
Roll the data stream onto the new template so new indices are correct, and backfill older indices only as far as your retention and query patterns justify. Anything about to age into the delete phase is not worth the I/O.

The short version

Dynamic mapping is a default, not a design. Decide per subtree: model it explicitly, make it a cheap keyword, flatten it, or drop it. Set ignore_dynamic_beyond_limit so the next schema surprise shows up as a metric instead of a page, and keep the field count in the same review where you look at shard sizes.

If you are already raising total_fields.limit to keep ingest alive, that is the moment to look at the mapping rather than the node count. Our cluster and cost review starts with exactly this measurement, and the contact form is the fastest way to describe your cluster to us.

Mapping explosion: field limits, dynamic templates, and when to reach for flattened