The usual way to add semantic search to Elasticsearch involves a service you own: read documents, call an embedding model, write vectors back into a dense_vector field, and repeat the same call at query time for the user's text. That service is not hard to write. It is annoying to operate — it drifts out of sync with the index, it has to be re-run whenever you change models, and it is one more thing on call.
semantic_text plus inference endpoints move that work inside the cluster. It is worth understanding precisely, because it removes real operational surface and adds a specific set of failure modes in exchange.
What an inference endpoint is
An inference endpoint is a named, reusable handle for a model. You create it once and refer to it by id:
PUT _inference/text_embedding/catalog-embed
{
"service": "elasticsearch",
"service_settings": {
"model_id": ".multilingual-e5-small",
"num_allocations": 2,
"num_threads": 1
}
}
The service is the part that matters for architecture. elasticsearch runs the model on ML nodes in your own deployment. You can instead point at an external provider — OpenAI, Cohere, Azure, Bedrock, Hugging Face — in which case the cluster makes outbound HTTPS calls on your behalf and your ingest throughput is governed by somebody else's rate limits. Both are legitimate; they have different blast radii, and you should pick deliberately rather than by whichever example you copied.
For sparse retrieval, ELSER is the same pattern with sparse_embedding instead of text_embedding. ELSER is English-first and produces weighted-token output that behaves more like a smart BM25 than like a dense model; for a single-language catalog it is often the easier starting point, because nothing about scoring becomes mysterious.
What semantic_text does
PUT products
{
"mappings": {
"properties": {
"name": { "type": "text" },
"description": { "type": "text",
"copy_to": "description_semantic" },
"description_semantic": {
"type": "semantic_text",
"inference_id": "catalog-embed"
}
}
}
}
Index a document with plain text in description and three things happen without you writing any of them: the text is chunked, each chunk is sent to catalog-embed, and the resulting vectors are stored in a nested structure under the field. Dimensions, similarity function, and index options are inferred from the endpoint — which is the point. The most common production bug in hand-rolled vector search is a mapping whose dims or similarity no longer matches the model that produced the vectors, and this field type makes that mismatch impossible to express.
Keep the original text field. The copy_to above is deliberate: you want lexical matching on description and semantic matching on description_semantic, because you are going to combine them.
Querying it
The short form does query-time inference for you:
GET products/_search
{
"query": {
"semantic": {
"field": "description_semantic",
"query": "waterproof jacket for hiking in cold rain"
}
}
}
In practice you want both retrieval styles and a fusion step. Retrievers express that without application-side merging:
GET products/_search
{
"retriever": {
"rrf": {
"retrievers": [
{ "standard": { "query": { "multi_match": {
"query": "waterproof hiking jacket",
"fields": ["name^3", "description"],
"minimum_should_match": "2<75%" } } } },
{ "standard": { "query": { "semantic": {
"field": "description_semantic",
"query": "waterproof hiking jacket" } } } }
],
"rank_window_size": 100,
"rank_constant": 20
}
},
"post_filter": { "term": { "in_stock": true } }
}
Same reciprocal-rank-fusion mechanics we covered in the hybrid search post — the difference is that no part of this pipeline lives in your application, so there is no second place for the model id to be wrong.
Where the cost and latency actually land
Three numbers decide whether this is a good idea for your workload.
Ingest throughput. Every document now waits on a model call. A small E5-class model on a couple of allocations handles hundreds of chunks per second, not tens of thousands. If you are reindexing 50 million documents, size the ML tier for the backfill, not the steady state, and expect the backfill to be the long pole in the migration. Bulk indexing that used to be disk-bound becomes inference-bound.
Query latency. Query-time inference adds a model call to every search — typically single-digit to low tens of milliseconds for a small local model, and whatever the network gives you for a hosted provider. Measure it at p99, not at the mean, and measure it with your real concurrency. If your search budget is 100 ms end to end, an external provider on a shared rate limit is a bad fit.
Memory and storage. Dense vectors are the expensive part of the index. A 384-dimension float vector is about 1.5 KB before overhead, and chunking multiplies that by chunks per document — a long description can easily produce five. Quantization is the lever: int8 HNSW cuts vector memory roughly fourfold and BBQ-style binary quantization much further, at a recall cost you should verify rather than assume. This is exactly what your judgment list and _rank_eval harness are for: turn quantization on, re-score, and decide with a number.
OpenSearch: the same idea, different names
OpenSearch has had this shape for a while under different vocabulary. You register a model with ml-commons, create an ingest pipeline with a text_embedding processor, map the target field as knn_vector, and query with the neural query — or hybrid with a normalization processor for fusion. Functionally comparable; more moving parts visible to you, and no single field type that hides chunking. If you are running both platforms, do not let the syntax gap talk you into believing one has capabilities the other lacks.
When not to do this
If your users search by part number, SKU, or exact title, semantic retrieval will mostly add latency and index size while shuffling results you already ranked correctly. If you have no judgment list, you cannot tell whether it helped — build that first; it costs a day. And if your embedding model is likely to change soon, note that changing it means a full reindex of the semantic field, because every stored vector came from the old one. That is not an argument against semantic_text; it is an argument for treating the endpoint id as a schema decision.
The honest summary: this is a reduction in operational surface, not a relevance win on its own. The relevance win comes from the same measured work it always did. What you get here is one fewer service to keep in sync with your index — which, on a small team, is worth a lot.