The same question returns different passages every run

Run the same question 20 times against an index nobody is writing to, and record the chunk ids that come back. If the top 2 or 3 hold steady while positions 6 through 10 shuffle, that is approximate search doing what it was built to do, and there is nothing to fix. If the top of the list moves, or if the results differ between users and sessions rather than between runs, something in front of the index is changing the query. Only the second class is a bug.

The two classes have no repair in common. One is a parameter you may choose to spend latency on. The other is a defect in the query path that keeps producing unexplainable behaviour long after you finish tuning the index. Deciding which you have takes an afternoon; guessing takes a fortnight.

The harness that settles it in an afternoon

The point of the harness is to remove every source of variation you did not intend to measure, one at a time, until the only thing left moving is the search itself. Run it as a script, not by hand in a console, because you will run it again after every change.

1. Fix the input as bytes. Not the user's question retyped, the exact string the application sent, including trailing whitespace and any conversation history the rewrite step folds in.
2. Pin the index. Stop ingestion for the duration or query a snapshot. An index that is being written to during the run makes every other measurement meaningless.
3. Capture the embedded vector, not just the query text. Hash it and record the hash on every run. If the hash changes between runs, you have stopped debugging retrieval and started debugging the rewrite step.
4. Run it 20 times in one process, and record the ordered chunk ids and their scores to 4 decimal places. Two decimal places hides the exact case you are hunting.
5. Compute the pairwise overlap of the id sets and, separately, count how many runs produced a different rank-1 id. Those two numbers point at different causes, which is why you keep them apart.
6. Repeat the whole run in a second process and against a second replica. Variation that appears only when you cross a process or a host boundary has a different cause from variation inside one process.
7. Repeat it a day later without pinning anything. A result that was stable under a pin and unstable in production tells you the corpus moved, not the search.

Reading the two numbers the harness produces

The variation that is the design working

Approximate nearest neighbour search exists because exact search over millions of vectors costs more than the answer is worth. Every approximate index buys its speed by not looking at most of the corpus, and what it chooses to look at depends on where it started and how much budget it had.

• Graph traversal width. An HNSW-style index walks a proximity graph from an entry point, keeping a candidate beam of a configured size. A narrower beam finds a slightly different, slightly worse neighbourhood — and the entry point itself can change after a merge.
• Cluster probing. An inverted-file index assigns vectors to clusters and reads only the few you probe. A chunk sitting near a cluster boundary is found or missed depending on which lists that particular query touched.
• Shard merging. With several shards each returning their own top-k, the global list is assembled from partial lists. A chunk ranked one place below the cut inside its shard cannot be recovered by the merge, and shard assignment changes when you rebalance.
• Quantised distances. Compressed vectors make scores approximate by construction, so two chunks whose true similarity differs in the fourth decimal can trade places between runs.
• Unstable sorts. Near-duplicate chunks — the usual product of generous overlap, described in chunk overlap and what it duplicates — produce genuine score ties, and a sort with no secondary key orders them arbitrarily.

None of these is a defect, and chasing determinism by widening the search until the list stops moving is the wrong response. The question that matters is not whether the ordering is stable but whether the passage that answers the question is in the candidate set at all, which is what recall@k measures. A perfectly repeatable search that never returns the right chunk is worse than a slightly jittery one that always does.

A stable result list is not a goal. It is a side effect of a search that had enough budget, and the useful question is whether the answer was in the list at all.

The variation that is a bug

Everything in this second list produces the same surface symptom and none of it is inherent to vector search. Each one is a thing your own system does to the query before or during the search.

• A generative rewrite above zero temperature. If a model expands or rephrases the question before the search runs, two runs are two different searches. The embedding hash from the harness catches this immediately, and what that step is actually for is set out in query rewriting before retrieval.
• An index accepting writes during the run. Fresh chunks change the neighbourhood, deletions leave tombstones, and a compaction in the middle of your test rewrites the graph under it.
• A filter that depends on the session. Permission groups resolved from a cache with a short expiry, a tenant identifier that silently defaults when a claim is missing, or a relative date window computed at request time all narrow the pool differently for the same words.
• Replicas that each built their own graph. Two hosts fed the same documents in a different order produce different approximate structures, so a load balancer becomes a source of randomness.
• A cache serving a fraction of requests from an earlier build. This is the nastiest of the five, because the results are internally consistent and simply belong to last week's corpus.
• A search budget being exceeded under load. An approximate index can legitimately return the best it found within its time or effort allowance rather than raising an error, so the failure arrives as a slightly worse list rather than as an alert.

The replica case is invisible in any environment small enough to run on one host. If you operate the index yourself — one of the less-discussed reasons teams choose to, alongside the arguments in private LLM deployment — you can pin a build, promote it explicitly, and refuse to serve two builds at once. If you do not operate it, ask the provider what a build identifier is and whether you can see it.

Record enough per request to skip the guessing next time

Every cause above is distinguishable from a single well-populated trace, and indistinguishable from a log line that records only the question and the answer. The fields are cheap and the discipline is the same one described in the anatomy of a span in an AI trace.

What unmeasured variance does to everything downstream

Flaky retrieval does not stay in the retrieval layer. It contaminates every diagnosis you attempt above it, because the input to the model was different on the run where the answer was wrong. You cannot conclude that the model answered from memory rather than from the passage if you cannot show that the passage was reliably in the payload. You cannot run a position sweep to test whether facts in the middle of a long context get missed if the set of passages is not the same set each time.

It also sets a floor under evaluation. If the same question over the same corpus produces a different candidate set on 3 runs in 20, a scored difference smaller than that between two release candidates is noise you have labelled as signal. Measure the variance once, record it next to your scores, and treat any smaller delta as unproven. Building the evaluation set itself, and choosing what gates a release, is a separate discipline.

The harness, the trace fields and a small page that replays a stored query against a named index build are a few days of work, and they convert this entire class of question from argument into measurement. That is the sort of unglamorous instrumentation we tend to insist on early in AI agents and automation work. Related failure modes sit alongside this one in retrieval and grounding, part of the wider engineering library.

• retrieval
• debugging
• vector search
• reproducibility