Vector search memory: quantization, rescoring, and the RAM bill nobody budgeted

Semantic search demos are cheap. The production bill arrives later, and it is almost always memory. A dense vector index behaves like any other Lucene structure until the HNSW graph and its vectors stop fitting in page cache — then every query starts hitting disk at graph-traversal random-access patterns, and p99 goes from 20 ms to seconds with no change in traffic. This post is the sizing math, the quantization options that fix it, and what each one costs you in recall.

First: the arithmetic that predicts your problem

Raw float32 vectors cost dims x 4 bytes per vector, per copy.

• 10M docs, 768 dims, float32: 10,000,000 x 768 x 4 = ~30 GB of vector data per replica copy.
• Same corpus at 1,536 dims: ~61 GB.
• Add the HNSW graph itself: roughly num_vectors x m x 2 x 4 bytes for the neighbour lists. At the default m: 16 and 10M vectors that is another ~1.3 GB — small next to the vectors, but it is not zero.

Now compare that against the RAM actually available for page cache: total node RAM, minus the JVM heap (cap it at ~31 GB for compressed oops), minus whatever the rest of the index needs. A 64 GB node with a 31 GB heap has roughly 30 GB left for everything off-heap — and that 10M-doc float32 index wants all of it. Two replicas on the same node, or any other index sharing the box, and you are already swapping into disk reads.

That single calculation, run before the project starts, prevents most vector-search incidents we get called about.

The lever: quantization

Quantization stores each vector dimension in fewer bits. Elasticsearch exposes this through dense_vector index types; OpenSearch through the k-NN engine settings (Lucene, Faiss, with scalar and product quantization). The levels and their memory factors:

Encoding	Bits/dim	vs float32	10M x 768
float32	32	1x	~30 GB
int8 (scalar)	8	~4x smaller	~7.5 GB
int4	4	~8x smaller	~3.8 GB
binary / BBQ	1	~32x smaller	~0.9 GB

In Elasticsearch, index_options.type of int8_hnsw, int4_hnsw, or bbq_hnsw turns these on at index time; the quantized form is what stays hot in memory, while the full-fidelity vectors remain on disk for rescoring. In OpenSearch the equivalents are scalar quantization (encoder: sq, FP16/int8) and product quantization on the Faiss engine, plus binary/disk-based modes depending on version. Check what your specific version supports before designing around it — this area has moved fast and the defaults changed more than once.

A mapping example:

PUT products
{
  "mappings": {
    "properties": {
      "embedding": {
        "type": "dense_vector",
        "dims": 768,
        "similarity": "cosine",
        "index_options": { "type": "int8_hnsw", "m": 16, "ef_construction": 100 }
      }
    }
  }
}

What quantization costs in recall — and how to buy it back

Compression is lossy, so approximate distances get noisier and the candidate list degrades. The standard repair is oversample and rescore: retrieve more candidates than you need using the cheap quantized distances, then re-score the top of that list against the full-precision vectors on disk.

"knn": {
  "field": "embedding",
  "k": 10,
  "num_candidates": 100,
  "rescore_vector": { "oversample": 3 }
}

Rules of thumb we start from, then verify:

• int8: recall loss is usually small enough to ignore at modest oversampling. This is the safe default; if you do nothing else on this page, move off raw float32 to int8.
• int4: noticeable loss without rescoring; with oversampling around 2-4x it typically comes back close to baseline, at some CPU cost per query.
• binary / BBQ: the biggest win and the biggest dependency on rescoring. It works well on high-dimension modern embedding models (1,024+ dims) and poorly on short, low-dimension vectors — there simply isn't enough signal left in one bit per dimension.

Note what the rescore step does to your I/O story: the full vectors live on disk and get read for the oversampled candidates. That is fine on NVMe and painful on network storage. If your data nodes are backed by throughput-limited remote disks, measure before assuming binary quantization saves you anything net.

Measure recall — don't infer it

Do not argue about quantization from first principles. Compute recall against exact search on your own corpus:

1. Sample 200-500 real query vectors.
2. Run each as an exact kNN query (a script_score over the full vectors, or knn with num_candidates set high enough to be effectively exhaustive on a small index). Save the true top-k doc IDs.
3. Run the same queries against the quantized index at your intended k, num_candidates, and oversample.
4. Recall@k = mean overlap of the two ID sets.

We target recall@10 of 0.95 or better for product search, and accept lower for use cases where the vector leg is one input into a fused ranking — if you are combining BM25 and kNN with RRF, a few borderline misses in the vector leg are frequently invisible in the final ranking. That is the trade-off to state out loud: the tighter your ranking depends on the vector leg alone, the less aggressively you should quantize.

Pair the recall number with an end-to-end relevance metric (nDCG@10 over a judgment list) so you can see whether a recall drop is actually reaching users. Recall is the intermediate; the judgment list is the outcome.

Knobs that matter more than people expect

• num_candidates is your latency/recall dial at query time; ef_construction and m are set at index time and require a reindex to change. Get m roughly right (16 is a fine default; 32 for high-recall demands) and tune num_candidates in production.
• Filters change everything. Filtered kNN narrows the reachable graph; on highly selective filters the search may effectively fall back to brute force, or return fewer results than you expect. Benchmark with your real filter selectivity, not unfiltered.
• Merging is expensive. Building HNSW graphs happens at segment merge; heavy reindexing on vector fields is far more CPU-hungry than the same volume of text. Budget the ingest window accordingly.
• Replicas multiply everything. Each replica needs its own copy in page cache. Replica count is a memory decision on vector indices, not just an availability one.
• Exclude vectors from _source retrieval in your queries (_source: false plus fields, or source filtering). Shipping 768 floats per hit back through the coordinating node is wasted bandwidth and JSON parsing.

A sizing procedure that works

1. Compute raw vector bytes: docs x dims x 4 x (1 + replicas).
2. Choose a target encoding and divide (4x for int8, 8x for int4, 32x for binary).
3. Require that the result fits in page cache with 30-40% headroom after heap — heap capped at 31 GB.
4. Build a 10% sample index, measure recall and p99 at your real filter selectivity.
5. Extrapolate, then re-measure at full scale before go-live. HNSW does not scale perfectly linearly; the measurement at 10% tells you the shape, not the answer.

When the answer is "don't use vectors here"

Worth saying plainly: if your queries are exact-ish — SKUs, part numbers, names — BM25 with decent analyzers will beat an embedding model and cost a rounding error in RAM. Vector search earns its budget on vocabulary mismatch, where users describe what they want in words your documents never use. Prove that class of miss exists in your judgment list first; then quantize hard enough that the infrastructure stays boring.

If your kNN latency got away from you, or the embedding project's hardware estimate came back three times what anyone expected, that is the kind of measurement work we do — get in touch with the corpus size, dimension count, and current node spec and we can usually tell you within a call whether it is a quantization problem or a design problem.