Kibana-based observability teams have spent a decade writing aggregation JSON by hand, or clicking Lens until it produces something close. ES|QL — the piped query language Elastic shipped as GA in 8.14 and has been extending since — changes the shape of that work. It is worth adopting for a specific set of jobs, and it is the wrong tool for others. This is where the line falls in production, with the trade-offs stated.
What ES|QL actually is
A piped language with its own compute engine. You start from a source, then chain commands:
FROM logs-nginx-*
| WHERE http.response.status_code >= 500
| STATS errors = COUNT(*) BY host.name, http.request.method
| SORT errors DESC
| LIMIT 20
Two things matter about that. First, it composes: each pipe transforms the rows the previous one produced, so a five-stage analysis reads top to bottom instead of nesting four levels of aggs inside a terms bucket. Second, it does not run on the classic search path. ES|QL has its own execution engine, which is why it can do things the query DSL cannot — and why some things the DSL does well are missing.
The same engine is available through the _query endpoint, so it is scriptable, not just a Kibana feature:
POST /_query
{ "query": "FROM metrics-* | STATS p95 = PERCENTILE(system.cpu.total.pct, 95) BY host.name | SORT p95 DESC | LIMIT 10" }
OpenSearch has its own piped language, PPL, with a similar shape and a different function set. If you run both platforms, do not assume queries port cleanly; the idioms transfer, the syntax does not.
Where it replaces your aggregations
Ad-hoc incident analysis. This is the strongest case. During an incident you are asking questions you did not anticipate — group by upstream, filter to one customer, compute a ratio, sort. In DSL that is a rewrite each time. In ES|QL it is appending a pipe. The iteration loop is the product.
Computed fields at query time. EVAL creates columns from existing ones without a runtime-field mapping round trip:
FROM logs-app-*
| EVAL duration_ms = event.duration / 1000000
| WHERE duration_ms > 500
| STATS slow = COUNT(*), p99 = PERCENTILE(duration_ms, 99) BY service.name
That is measurable work you would otherwise do with a painless runtime field or, worse, in a spreadsheet after exporting.
Multi-stage aggregation. ES|QL lets you STATS and then STATS again over the result — per-host averages rolled up to a per-cluster median, for instance. Pipeline aggregations can express some of this, awkwardly; most teams gave up and did it client-side.
Enrichment during analysis. ENRICH joins an enrich policy's lookup data onto your rows mid-query, so you can group log lines by team owner or region without having added that field at ingest time. That removes a common reason for reindexing.
Alert rules that were previously unwritable. Kibana supports ES|QL-based rules, which means a threshold rule can be a real query — a ratio, a multi-stage stat — instead of a single-metric comparison plus a mental asterisk.
Where it does not belong
Relevance-ranked search. ES|QL has gained text search capability, but if your job is scoring documents for a user-facing product search — multi_match with field weights, minimum_should_match, phrase boosts, function_score, kNN with score fusion — the query DSL is the mature surface and stays that way. Do not port a tuned relevance query into ES|QL because it reads nicer.
High-QPS application queries. ES|QL is an analytics path. Result caching, request caching behavior, and the tuning knobs you rely on for a 20 ms p99 search endpoint are not the same. Benchmark before you put it in a hot request path; the honest default is "use DSL for the app, ES|QL for humans and dashboards."
Anything crossing very large row counts carelessly. Which brings us to the part that bites.
The cost and stability trade-off
The DSL makes you declare the shape of the work up front; ES|QL makes it easy to ask for an enormous amount of it in one line. Three habits keep that from becoming an incident:
- Filter before you aggregate. Put
WHEREon indexed fields as early in the pipe as possible, and always constrain@timestamp.FROM logs-*with no time bound scans your entire retention window across every tier. - Be specific in
FROM. Index patterns are cheap to type and expensive to run.FROM logs-nginx-*beatsFROM logs-*by whatever multiple your log estate happens to be. - Know the row limits. ES|QL applies internal ceilings on rows returned and processed — there is a default result limit and a configurable maximum above it. Hitting one produces truncation or an error rather than a correct answer; treat an unexpected limit warning as a signal that the query is wrong, not as something to raise the setting for.
Operationally, watch GET _tasks?actions=*esql* during rollout and keep an eye on circuit breaker trips on data nodes. Analysts iterating in Kibana against a frozen tier can generate far more I/O than the same people clicking through Lens did, because the loop is faster.
How we introduce it
- Pick observability first, search last. Logs and metrics analysis is where the win is unambiguous.
- Keep existing dashboards. Do not convert working Lens panels wholesale. Add ES|QL panels for the questions that had no panel.
- Write three real queries with the on-call team — the top-error breakdown, the latency percentile by service, the enrichment join — and put them in a runbook. Adoption follows from examples people recognize.
- Set a query policy in writing: time bound required, explicit index pattern, no ES|QL in application request paths without a benchmark.
- Re-check version support. Function coverage, joins, and search capability have moved fast across recent minors. Confirm what your deployed version supports rather than what a blog post from last year described.
The short version
ES|QL removes a category of friction that had nothing to do with your data: the gap between having a question and expressing it. That is worth real money on an observability team measured in mean time to understanding. It does not replace the query DSL for ranked search or for latency-sensitive application queries, and it makes it easier than before to ask for an expensive scan by accident. Adopt it for the analysis loop, bound it with a time filter and a narrow index pattern, and keep your tuned search queries where they are.
If your team is standing this up across a large log estate — or deciding between ES|QL on the Elastic Stack and PPL on OpenSearch — that is the kind of question our observability pipelines work starts from: measure the current query load first, then change it.