What a reranking stage does to a candidate list
In short
A reranker is a second scoring pass over a candidate list the first stage already produced. It reads the query and each passage together rather than comparing 2 vectors computed separately, which lets it notice negation, qualifiers and exact terms that a stored embedding averaged away. It reorders. It cannot add a passage the first stage never fetched.
Key takeaways
- A reranker changes the order of a candidate list and never its membership. That is the whole contract.
- It reads query and passage together, so it can weigh negation and qualifiers a stored vector flattened.
- First-stage recall is a hard ceiling. Rerank 50 candidates and you inherit whatever recall@50 was.
- Its scores are per-query artefacts. Thresholding on them across different queries needs calibration.
- A reranker is not a filter. Reordering a list containing passages a user may not see is still a leak.
A reranker is a second, more expensive scoring pass over a candidate list that already exists. The first stage fetches perhaps 50 passages cheaply; the reranker scores each against the query and hands back the same 50 in a different order, from which you send the top few. It changes order, never membership.
The 2-stage shape exists because the 2 jobs have different cost curves. Finding 50 plausible passages among 5 million must be sublinear, which forces an approximate index and precomputed vectors. Ranking 50 things well can afford a model call per candidate.
Why a joint pass sees what a stored vector cannot
First-stage vectors are computed independently and in advance. The passage was embedded at index time, months before anyone asked this question, so its vector must be a summary useful for every query that might ever arrive. The query is embedded separately, and the comparison is between 2 fixed-length summaries that never met.
A reranking model reads query and passage together in a single pass, so nothing is summarised in advance. It can weigh the exact term the user typed, the qualifier attached to the rule, and the negation.
Ask "does this policy apply to contractors" against a passage reading "this section does not apply to contractors". The 2 texts share almost all their vocabulary, so their independently computed vectors sit close together and the first stage is right to return it. Whether the passage confirms or excludes is a distinction a joint pass can represent and 2 precomputed summaries structurally cannot.
What it can change, and what it cannot
| Property | First stage | Reranking stage |
|---|---|---|
| Set of passages considered | Chooses it, from the whole corpus | Inherits it, and cannot add to it |
| How the query meets the passage | 2 vectors computed separately, compared by distance | Both read together in one pass |
| Cost per query | One index lookup, near-constant with corpus size | One model call per candidate, linear in candidates |
| Scale it operates at | Millions of passages | Tens, occasionally low hundreds |
| Cost of a mistake | A missed passage is unrecoverable | A misordering is visible and correctable |
The ceiling it inherits
Because membership is fixed before the reranker runs, the first stage sets a hard ceiling on what it can achieve. If the answer-bearing passage was in the top 50 for 72 of 100 questions, reranking those 50 delivers at most 72 answerable questions, however good it is. That ceiling is recall@k at the depth you rerank from.
Which gives the 2 numbers worth reporting side by side: recall at the candidate depth, and recall at the depth you send after reordering. The first is the ceiling, the second the delivery, and the distance between them is what the stage is being asked to close.
A reranker is an editor, not a researcher. It improves the order of what was brought back and has no way to fetch anything else.
Where it sits relative to filters and fusion
Three things operate on candidate lists and are routinely confused. A filter decides admissibility before scoring — a hard rule about which passages may compete at all, which is why the difference between filtering and ranking repays being pedantic about. Rank fusion merges 2 orderings, typically a keyword list and a vector list. Reranking rescores a single list.
The consequence that matters most: a reranker is not a substitute for a filter. Demoting a passage the user is not entitled to see leaves it in the candidate set, one configuration change from the top. Entitlement is admissibility, enforced before scoring.
Nor does reranking make an answer supported. Better-ordered evidence raises the odds that the backing passage is in front of the model; whether a claim actually rests on supplied evidence is a separate property, set out in groundedness as a property of an answer.
The operational shape of adding one
A reranking stage is a model you run on every query, once per candidate, which makes it the first part of your retrieval path with a per-request inference bill and a serving decision attached. Whether it runs behind someone else's endpoint or on hardware you operate has thresholds of its own, in the hosted API and self-hosted inference threshold. Whether it earns its place at all is a comparison this cluster handles separately.
What makes it maintainable is smaller than the model choice: a page showing the candidate list before and after reordering, with both scores, for any query someone can paste in. Without it, every argument about ranking runs on memory. It is a day of work, and we treat that as part of the build in internal tools and operations software. Surrounding terms sit in retrieval and grounding, part of the engineering library.
Frequently asked questions
Short answers to the follow-ups this page tends to raise.
What is a reranker in a retrieval pipeline?
It is a second scoring stage that reorders the candidate passages the first stage returned. It scores each candidate against the query directly and sorts by that score; the few at the top reach the model. The candidate set is fixed before it runs.
What is the difference between a cross-encoder and a bi-encoder?
A bi-encoder embeds the query and the passage separately, so the passage vector is computed long before the query exists and has to serve every possible question. A cross-encoder reads both together and scores that specific pair. The first is fast enough for millions of passages; the second is accurate enough to be worth running on a few dozen.
Can a reranker retrieve documents the first stage missed?
No. It only sees the candidate list handed to it, so a passage absent from that list cannot be promoted, scored or cited. If good passages are missed entirely, the repair is a wider candidate set, a lexical arm or different chunking.
Does a reranker replace metadata filtering?
No, and treating it as one creates a permission bug. A filter removes passages from consideration; a reranker moves them down while leaving them in. Anything driven by entitlement or tenancy belongs in the filter applied inside the search, where a later config change cannot promote it back into an answer.
- reranking
- retrieval
- ranking
- definitions
The work behind this page
Builds from our portfolio that this page draws on.
AskVault
An AI internal knowledge-search platform that answers employee questions from your own docs — grounded in citations, with knowledge gaps surfaced and deflection tracked.
Productivity AIReply Rail
AI-drafted Google, Yelp, and Facebook review responses for local businesses — one queue, one click.
Local Business AIRead next
- Recall@k, and why it is the number retrieval lives or dies byRecall@k is the share of questions whose answer-bearing passage made the top k. It gates everything downstream, because no reranker or prompt recovers a passage that was never fetched.definition
- Chunk overlap: what it protects against and what it duplicatesOverlap is insurance against a chunk boundary landing in the middle of one idea. The premium is paid in duplicated candidates crowding a fixed number of prompt slots.definition
- You know the document is indexed and it still never comes backFour layers can quietly drop one document, and they look identical from the outside. Probe them in cost order — existence, filter, exact text, rank — and each probe eliminates exactly one.diagnostic
- The right passages were retrieved and the answer is still wrongIf the answer-bearing passage was in the context and the answer is still wrong, retrieval tuning cannot help. Four synthesis defects produce this, and each leaves its own mark in the output.diagnostic
- It worked on five hundred documents and broke at fifty thousandNothing regressed when the corpus grew. A candidate budget that comfortably held the answer at 500 documents now competes against a hundred times as many near-neighbours, and k never moved.diagnostic
- Adding one new source made unrelated answers worseNothing about the old passages changed. A new source ranks moderately well on a great many queries, and a fixed top-k has to give it those slots by taking them from something else.diagnostic
Working on something in this space?
Tell us where you are in a sentence or two. We'll tell you honestly whether we're the right team, and what a sensible first slice of the work looks like.
Start the conversation