Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Search and Vector Embeddings: Two Indexes, One Recall Problem

Overview

Keyword search and vector search are usually taught as different subjects — information retrieval versus machine learning — but they are the same economic transaction executed twice: spend expensive computation at write time to build a structure that makes query time cheap. The inverted index precomputes term→document postings; the vector index precomputes neighborhoods in an embedding space. Both then face the same lifecycle problems: the corpus changes under the index, the structure is approximate or lossy in specific ways, and quality can silently degrade because nothing in either system raises an error when it gets worse. This page maps the full vector pipeline — documents → chunking → embedding model → approximate-nearest-neighbor (ANN) index → reranker → cache → evaluation loop — as a connected system, names where it shares invariants with the inverted-index world (write-time amortization, segment merges, freshness versus cost) and where it breaks from it (probabilistic recall, a versioned vector space that makes partial reindexing toxic), and works through the two propagation failures that dominate production incidents: model drift between index and rebuild, and stale embeddings after document updates.

The Nodes, Independently

Documents and chunking. The ingestion path splits documents into chunks — paragraphs, sections, fixed token windows with overlap — because the embedding model has a context limit and because retrieval granularity should match answer granularity. Chunking is the most underrated decision in the pipeline: it defines the unit the vector space will represent, and a change in chunking changes every vector in the index, invisibly. Chunk size trades context against precision — a 512-token chunk retrieves about its subject; a 64-token chunk retrieves the sentence — and overlap trades storage against the risk of an idea straddling a boundary. Structured metadata (title, section, URL, timestamp) should ride alongside every chunk, because filters and citations need it even though the model does not.

Embedding model. The model maps text to a dense vector (say 768 or 1,536 dimensions) such that semantic similarity in the training distribution becomes geometric proximity. Two properties matter architecturally: the vector space is defined only by a specific model version — vectors from model v1 and v2 are coordinates in unrelated spaces — and the model is a lossy compression of everything it saw, which is why embeddings generalize gracefully inside their distribution and fail quietly outside it.

ANN vector index. Exact nearest-neighbor over 30M vectors is hopeless at query time, so the index approximates: HNSW navigable small-world graphs or IVF clustered postings trade a tunable recall loss for orders-of-magnitude speedup. Recall@k becomes an engineering parameter — probe counts, ef-search, list sizes — with a latency/memory/recall triangle, exactly like the tuning knobs of any physical index.

Reranker. A cross-encoder scores the top-k candidates jointly (query and document in one pass), which is far more accurate than bi-encoder similarity and far more expensive — so it runs on 10–100 candidates, not 30M documents. It is a precision instrument bolted onto a recall instrument.

Cache and eval loop. Query embeddings, ANN results, and full responses are cached keyed by query text (and model version); the evaluation loop runs golden query sets through the pipeline and tracks recall@k, nDCG, and latency — the only component that can see the silent failures the rest of the system produces. Caching has the usual two-sided contract: a response cache must be invalidated when the index changes under a query, which is why serious deployments key result caches by index version rather than trusting TTLs alone — a stale result after reindex is the stale-embedding failure relocated one layer up. The eval loop is also the pipeline’s regression suite: ANN library upgrades, quantization changes, and chunker edits are all “refactors” that can move quality, and only a frozen golden set turns “the answers feel worse” into a diff.

How the Nodes Feed Each Other

The write path flows documents → chunker → embedding model → ANN index build; the read path flows query → query embedding → ANN search → reranker → cached response. The paths meet only inside the index, which is why every cross-path failure is a version failure: the write path executed with model v1 and chunking scheme A, and the read path executes with model v2 and chunking scheme B, and nothing errors — the geometry just stops meaning anything. The eval loop closes the only feedback edge: golden sets detect drift in quality, which propagates back into decisions about chunking, model choice, and ANN parameters. Cost flows the other way: index builds are batch jobs amortized over millions of queries, reranking is per-query spend concentrated on the candidates most likely to matter, and cache hits are free — a cost gradient that mirrors any write-optimized store.

The Inverted-Index Comparison, Precisely

Because the sibling map (search-inverted-index.md) covers the keyword side in depth, the precise correspondence is worth stating rather than gesturing at. The analyzer (tokenize, stem, stop-word) corresponds to the chunker-plus-embedder: both define the representation that query time and index time must agree on, and both silently wreck retrieval when changed on one side only. The postings list corresponds to the ANN neighborhood structure: both are write-time structures that turn a scan into a lookup. BM25 scoring corresponds to bi-encoder cosine similarity: both rank by a cheap precomputable approximation of relevance. The segment merge corresponds to incremental index build: both batch fresh content into searchable units at a chosen freshness/cost point. The correspondences break in two places that matter. First, exactness: a postings lookup finds all documents containing a term, while ANN finds most of the nearest neighbors — recall is a dial, not a promise. Second, the representation contract is stronger on the vector side: an analyzer change invalidates only the affected terms’ postings, while an embedding-model change invalidates every vector in the index, which is why versioned indexes are mandatory in one world and merely nice in the other.

The Shared Invariants

Write-time amortization. Both indexes exist because query time is the scarce resource; both therefore inherit “the index is a projection of the corpus, and it ages.” Segment-merge economics. Fresh documents land in small segments (or appended vectors) that are periodically merged — the LSM pattern — so freshness, write amplification, and read amplification trade against each other in both worlds. Compatibility across versions. The inverted index’s analyzer chain must match between index time and query time or terms stop matching; the vector index’s model version must match between build time and query time or coordinates stop corresponding. Both are the same invariant: the pipeline’s read and write sides must agree on the representation contract. Approximation must be measured. B+tree lookups are exact; BM25 ranking is deterministic-but-scored; ANN recall is probabilistic. Every layer downstream of an approximation inherits its error, so recall@k is a system property that must be measured, alerted on, and re-measured after any parameter change.

Failure Propagation

Model drift between index and rebuild is the characteristic silent failure. A team retrains the embedding model, reindexes 30% of the corpus for a pilot, and runs both populations in one index: v1 vectors and v2 query embeddings are points in different spaces, so cross-population similarity is noise. Nothing throws; latency is unchanged; nDCG on the golden set quietly drops — if a golden set exists. The fix is structural, not operational: versioned indexes (index-v23), full dual-serving during rebuild, cutover by evaluation, never a mixed population. Stale embeddings after updates are a cache-coherence failure wearing ML clothes: a document’s price or policy text changes, the embedding is not recomputed, and the system confidently retrieves the old meaning. Propagation is invisible because retrieval succeeds — the answer is just wrong, and users, not monitors, discover it. Chunking drift propagates the same way: an upstream format change alters chunk boundaries, embeddings remain “valid,” and quality sags below any alarm threshold. ANN degradation under pressure is the capacity failure: memory growth forces parameter downgrades (smaller ef-search, fewer probes), recall drops, rerankers receive worse candidate pools, and the quality loss compounds down the read path while every SLA stays green. Cache-layer staleness completes the set: response caches keyed only by query text serve pre-reindex results after a rebuild, so even a perfectly reindexed index answers with yesterday’s winners until TTLs expire — which is why index version belongs in the cache key, next to the model version. The pattern across all five is uniform: each failure crosses a layer boundary (write path to read path, index to cache, ANN to reranker), and each hides because every individual layer reports health.

Design Levers

Versioned indexes and dual-serving turn model upgrades from terrifying to routine: build index-v(n+1) alongside v(n), shadow-evaluate with real query traffic, cut over like a blue-green deploy, keep the old index for instant rollback. Hybrid keyword plus vector retrieval is the highest-value structural lever: BM25 catches exact identifiers, error codes, and part numbers that embeddings blur; vectors catch paraphrase and intent that keywords miss; reciprocal-rank fusion combines them. Most “vector search failed” war stories are really “we deleted the keyword index” stories. Metadata filters need a placement decision: pre-filtering inside the ANN traversal preserves recall but couples the index to filter selectivity, while post-filtering can return fewer than k results. Content-addressed embedding caches keyed by (model version, text hash) make reindexing incremental and make the model version an explicit part of every cache key — the same discipline as build-cache keys.

Continuous evaluation with canary query sets, drift alarms on recall@k, and per-release scorecards is the only instrument that reads the silent failures; budget it like an SLO, not a hobby. Incremental reindexing keeps the pipeline fresh without full rebuilds: append new chunks to a small hot segment (near-real-time, exactly like search engines), merge on a schedule, and reserve full rebuilds for model or chunker changes — the decision of what triggers a full rebuild versus an incremental update is the vector-pipeline equivalent of choosing what invalidates a cache. Backfill tooling matters as much as the index: resumable, rate-capped chunk-embedding jobs with checkpointing, because a 4-hour build that dies at hour 3 without checkpoints is a 4-hour build again. Quantization and dimension choices are the capacity levers — int8 or product quantization trades a measured recall loss for 4–32× memory reduction, which in turn buys headroom for dual-serving during versioned rebuilds.

A Concrete Worked Example

Corpus: 10M documents, chunked at ~512 tokens with 50-token overlap → 30M chunks. Embeddings at 768 dimensions, float32: 30M × 768 × 4 bytes ≈ 92 GB raw; an HNSW graph adds neighbor links for roughly 1.5–2× total memory, so plan on 140–190 GB before replicas. Re-embedding at 2,000 chunks/s: 30M ÷ 2,000 ≈ 4.2 hours of pure embedding compute before index build even starts — which is why dual-serving during a model upgrade needs capacity for two indexes (~300–380 GB) for that window, and why int8 quantization (4× smaller) is usually adopted before model upgrades rather than after. Query path: ANN top-100 in 5–10 ms p99, cross-encoder rerank of top-10 in ~50 ms, 80–90% of queries served from the response cache at peak. Now the incident: a chunking change ships on Tuesday; embeddings still compute fine; recall@10 on the golden set drifts from 0.91 to 0.78 over the week. No error, no alarm — except the eval loop, which is the one node that noticed, named the cause, and gated the rollback.

Interview Questions

  1. Why can you never mix vectors from two embedding-model versions in one index, and what deployment pattern makes model upgrades safe? Name the resource cost of that pattern for a 30M-vector corpus.
  2. Your vector search quality degraded 15% with zero errors and unchanged latency. Enumerate the propagation paths that degrade quality silently, and say which telemetry would have caught each.
  3. When does hybrid BM25-plus-vector beat either alone? Give a concrete query class where the vector index fails and the inverted index succeeds, and vice versa.
  4. Where do you put metadata filtering — before, inside, or after ANN traversal — and what does each choice cost in recall, latency, and index design?
  5. Contrast freshness handling in an inverted index (near-real-time segment refresh) with a vector index after 1M document updates. What makes the vector case harder?

Key Takeaways

  • Keyword and vector search are one economics — write-time computation amortized against query-time cost — with the same segment-merge and freshness trade-offs.
  • The vector space is versioned: model version is part of the representation contract, and partial reindexing across versions produces confident nonsense rather than errors.
  • The pipeline’s dominant failures are silent — model drift, stale embeddings, chunking drift, ANN parameter downgrades — so the evaluation loop is not a nicety but the only working sensor.
  • Hybrid keyword-plus-vector retrieval is the default architecture: exact-match and paraphrase coverage are complements, not competitors.
  • Treat index upgrades as blue-green deployments with shadow evaluation and rollback, and treat embedding caches as build caches keyed by content and model version.

Cross-References