Ingest is usually the part of an observability build that gets designed. Alerting is the part that accretes. Someone adds a threshold rule off a dashboard during an incident, nobody removes it, and eighteen months later the on-call channel gets 300 messages a week and the team has learned to ignore all of them. That is worse than no alerting, because it costs money and buys false confidence.
This is how we approach alerting rules in Kibana and in OpenSearch Alerting when the rule set has to survive contact with a real on-call rotation.
Start from the page, not the metric
One question decides whether a rule should exist: if this fires at 03:00, what does the person do? If the answer is "look at it in the morning," it is a dashboard or a ticket, not a page. If the answer is "nothing, it clears itself," delete it.
A workable split:
- Page — customer-visible symptom, degrading now, human action shortens it.
- Ticket — real but not urgent: a disk that will fill in nine days, an ILM policy stuck in an error step.
- Dashboard only — context you consult during an incident, like per-node CPU.
Most noisy rule sets are cause alerts on the page tier. Cause alerts (heap at 80%, CPU at 90%, one node dropping out) fire constantly under normal operation. Symptom alerts (checkout search p99 above 800 ms, ingest lag above 10 minutes, 5xx rate above 2%) fire when users are hurting. Page on symptoms; keep causes as context linked from the alert.
Picking the rule type
Kibana gives you several rule types and they are not interchangeable:
- Elasticsearch query rule (KQL/DSL or ES|QL) — the general tool. Good when the condition is a count or an aggregation over a filtered slice. ES|QL rules let you express the whole condition, including stats and eval, in one statement.
- Index threshold — a fine, cheap choice for "metric X over group Y crosses N". Don't reach for a custom query when this does the job.
- APM latency/error-rate and custom threshold rules — service-aware; they already know about service, environment, and transaction type, so grouping is one field instead of a hand-written terms agg.
- SLO burn rate — the one you want for user-facing reliability; see below.
OpenSearch Alerting has the same shapes under different names: per-query, per-bucket, per-document, and cluster-metrics monitors. Per-bucket monitors are the important ones — they alert per group (service, tenant, index) rather than firing one alert for a whole aggregation.
Look-back windows and ingest lag
The most common broken rule we find is a correct condition over the wrong window. A rule that checks the last 1 minute against data that arrives with 90 seconds of pipeline lag evaluates an empty window and either never fires or fires constantly, depending on how the condition is written.
Measure the lag before you set the window. The cheap version: compare event time to index time.
POST logs-*/_search
{
"size": 0,
"runtime_mappings": {
"lag_ms": {
"type": "long",
"script": "emit(doc['event.ingested'].value.toEpochMilli() - doc['@timestamp'].value.toEpochMilli())"
}
},
"aggs": { "lag": { "percentiles": { "field": "lag_ms", "percents": [50, 95, 99] } } }
}
Then set the rule's look-back to roughly p99 lag + evaluation interval + margin. For a pipeline with p99 lag near two minutes and a one-minute check interval, a five-minute window is honest; a one-minute window is a lie. Use a delay/offset where the rule type offers one so you are evaluating a window that is actually complete.
The cost side matters too. A rule checking every 30 seconds over a 24-hour window across logs-* is a heavy recurring query. Look-backs should be the smallest window that is complete, and check intervals should match how fast you could possibly respond.
Burn-rate alerts instead of static thresholds
Static latency and error thresholds have a structural problem: whatever number you pick is wrong at some traffic level. Pick it tight and you page on a two-minute blip; pick it loose and a slow bleed burns the whole month's error budget without firing.
SLO burn rate fixes this by alerting on how fast you are consuming the error budget rather than the instantaneous rate. Define the SLO — say 99.9% of search requests under 800 ms over 30 days — and then alert on multi-window burn:
- Fast burn: 14.4x budget consumption over 1 hour, confirmed by a 5-minute window. At that rate the 30-day budget is gone in about two days. That pages.
- Slow burn: 6x over 6 hours, confirmed over 30 minutes. That is a ticket.
The two windows matter: the long window gives significance, the short window makes the alert recover quickly once the problem stops. Kibana's SLO feature implements this directly with burn-rate rules; on OpenSearch you build the same shape with a per-query monitor over a good/total ratio and two triggers.
The practical effect is that low-traffic services stop paging on three failed requests, and a 0.4% error rate that would sit under a 1% threshold forever still gets caught, because it burns the budget.
Grouping, dedup, and throttling
A rule that fires once for "some service is bad" is useless during a partial outage. Group by the field the responder will act on — service.name, kubernetes.namespace, data_stream.dataset, tenant — and let the rule track one alert per group. In Kibana that is the group-by in the rule definition; in OpenSearch it is a per-bucket monitor.
Then tame the volume:
- Require N consecutive breaches before firing. Most single-interval spikes are noise or a scrape gap.
- Set an action frequency — notify on status change, not on every check — so one ongoing problem is one message plus a recovery, not 400 messages.
- Use a recovery action. An alert that never announces it cleared trains people to ignore the channel.
- Cap the group count. A rule grouping by a high-cardinality field can generate thousands of alerts from one upstream failure; bound it, and alert separately on "too many groups firing," which is itself the better signal.
Don't alert on a cluster you can't alert from
If the rules that watch production live in the same cluster as production, a cluster outage takes your alerting with it. Two mitigations, in order of effort: run the alerting rules on a small separate monitoring cluster using cross-cluster search against production, and add one external dead-man's-switch — a heartbeat rule that pages if it has not heard from the pipeline in 15 minutes, evaluated somewhere else entirely. The heartbeat is the single highest-value alert in most setups, and it is almost always missing.
Measure the rule set like a service
Alerting rules are code that runs forever, so review them on data, not vibes. Kibana writes alert documents to .alerts-* and rule execution results to the event log; OpenSearch keeps alert history indices. Build one dashboard over those with four numbers per rule:
- Fire count per week.
- Median time to recovery — rules that recover in under two minutes are mostly noise.
- Acknowledged vs auto-resolved ratio — auto-resolved and never touched means nobody acts on it.
- Execution failures and duration — rules that time out are silently not protecting you.
Then apply a policy: any rule that fired more than ten times in a month with no human action is demoted to a ticket or deleted. Any rule that has never fired in six months gets tested, because untested rules are usually broken rules. Shadow-run new rules to a silent channel for a week before they get to page anyone.
A reasonable starting set
For a team running the Elastic Stack or OpenSearch for their own services, the paging tier is small:
- Burn-rate rules on two or three real SLOs (availability and latency of the user-facing path).
- Ingest heartbeat / dead-man's switch per critical data stream.
- Cluster health red, and disk above the high watermark on any data node.
- One rule per known, previously-seen failure mode with a written runbook link in the action body.
Everything else is a ticket. If that list looks short, that is the point: a channel with four kinds of message in it is a channel people read.
It depends
Two trade-offs worth stating plainly. Multi-window burn-rate rules are more expensive to evaluate than a threshold, because every check runs two aggregations over long windows — on a large cluster, budget for that or pre-aggregate with a transform or downsampled index. And tightening rules reduces noise at the cost of detection time: requiring three consecutive breaches adds up to three intervals of delay. Choose the delay you can live with per rule, deliberately, and write the number in the rule description so the next person knows it was a decision.
If your on-call channel is noisy and you are not sure which rules are earning their place, the execution data usually answers it in an afternoon. That review is part of our observability pipelines work — get in touch with what your current rule set looks like.