Most ingest architectures were never designed; they accreted. Filebeat was there first, a Logstash box appeared to fix a parsing problem, an OTel collector arrived with the tracing initiative, and now three shippers with three configuration idioms feed one cluster. It works — until you need to change retention, add a stream, or explain the bill. Here is how we think about designing the pipeline on purpose.
The three collectors, honestly
Elastic Agent (with Fleet) is the right default in Elastic-centric estates. One agent per host replaces the zoo of Beats, integrations package dashboards and pipelines together, and Fleet gives you central policy management — agent config changes without touching hosts. Its gravity is toward the Elastic ecosystem: that's a virtue if you live there and a constraint if you might not always.
Logstash is no longer the mandatory middle tier it was in classic ELK, but it remains the strongest transform engine of the three: rich filter plugins, multiple outputs with independent delivery, and a persistent queue on disk. Modern architectures use it selectively — as a central transform/fan-out tier where that is genuinely needed — rather than as a required hop for every event.
The OpenTelemetry Collector is the vendor-neutral play and the natural choice where traces matter, where multiple backends must be fed, or where you want instrumentation and collection decoupled from any one vendor. Elastic and OpenSearch both ingest OTLP happily. Its transform capabilities have matured, but its ecosystem's center of gravity is signals-from-applications; host-level log collection is where the Elastic Agent integrations remain more turnkey.
The combinations are legitimate: OTel for application telemetry with Elastic Agent for infrastructure is a common and sensible pairing. What is not sensible is three overlapping collectors doing the same job on the same hosts.
Where transforms belong
Every event can be reshaped at the edge (agent/collector processors), in a middle tier (Logstash or an OTel gateway), or in the cluster (ingest pipelines). The rule of thumb:
- Drop early. Filtering noise at the edge is the only place it saves both network and cluster cost. Debug-level logs nobody will query should die on the host.
- Enrich centrally. Lookups and joins (adding owner, environment, or geo data) belong in a middle tier where reference data lives in one place, not on ten thousand hosts.
- Normalize at the door. Elasticsearch ingest pipelines are the cheapest operational home for final field renames and type coercion — versioned with the cluster, visible in one place, applied per data stream.
The most expensive anti-pattern is the inverse: shipping everything raw and parsing in the cluster, which spends your most expensive compute on your least valuable events.
Does a buffer earn its keep?
A queue between collectors and the cluster — usually Kafka — buys you two things: a shock absorber for ingest spikes and cluster maintenance windows, and a replay source for reprocessing after a bad parse or a schema change. Those are real benefits, and for large estates (roughly: hundreds of GB per day, multiple consuming systems, strict loss budgets) they justify the operational cost of running Kafka.
Below that scale, be skeptical. Elastic Agent and the OTel Collector both retry with local backpressure, Logstash has a persistent queue, and a cluster sized with modest headroom rides out normal spikes. A Kafka tier added "for safety" to a 50 GB/day estate is another distributed system to page someone about.
Schema discipline is the multiplier
Whatever the topology, pick one field vocabulary — ECS in Elastic-centric estates, OTel semantic conventions in OTel-centric ones — and enforce it at ingest. A service.name that means the same thing in logs, metrics, and traces is what makes cross-signal dashboards and alert correlation possible at all; without it every new stream ships with its own dialect and every dashboard grows another layer of per-stream special cases. The mapping from convention to convention is well-trodden; the discipline of applying it is what most estates are missing.
A worked default
For a mid-size estate that lives on the Elastic Stack, our default sketch is: Elastic Agent everywhere for infrastructure and logs, managed by Fleet; OTel SDKs and a small collector gateway for application traces and metrics; edge-level drop rules for known noise; enrichment in the gateway; final normalization in per-stream ingest pipelines; no Kafka until the numbers demand it; ECS everywhere. Data streams with ILM policies per stream complete the picture — the pipeline decides what arrives, the lifecycle policy decides what it costs to keep.
Design it once, write it down, and the next stream someone adds has a place to land — instead of a new box on a diagram nobody owns.