Query rewriting: what happens to the user's words before the search runs
In short
Query rewriting is a stage between the user's question and the search, and it is asked to do 3 unrelated jobs: resolve what a follow-up refers to, widen vocabulary the corpus may phrase differently, and reshape a question into text that looks like the passage answering it. Each has its own failure mode, and a rewrite nobody logs is a retrieval nobody can reproduce.
Key takeaways
- Three different operations share the name. Decide which one you need before adding any of them.
- Context resolution is the only rewrite most conversational assistants genuinely cannot do without.
- Rewriting an exact identifier destroys the one signal that would have matched it. Pass those through.
- Log the original and the rewritten query together, or every complaint becomes unreproducible.
Query rewriting is any transformation applied to what the user typed before it reaches the index. The word covers 3 distinct operations sharing nothing but their position in the pipeline: resolving a question that only makes sense in context, expanding it to cover vocabulary the documents actually use, and converting it into a form resembling the passages being searched.
They get conflated because they live in the same function, and that is expensive. Each solves a different problem and fails differently, and a team that adds all 3 because a framework bundles them has built 3 new places for an answer to go wrong.
Three jobs wearing one name
| Job | What the user typed | What reaches the index |
|---|---|---|
| Context resolution | And what about the second one? | The termination notice period in the second supply agreement, with the agreement named explicitly |
| Vocabulary expansion | PTO carryover rules | The same question plus annual leave, holiday entitlement and carry forward, because the handbook never says PTO |
| Form conversion | How do I reset a locked badge? | A short declarative passage describing badge reset, embedded and used as the query |
The third row is what is usually called hypothetical document embedding: have a model draft a plausible answer, embed the draft, and search with that instead of the question. A stored passage was written as a statement rather than a question, so a statement is a closer neighbour. It helps where questions and documents use genuinely different registers, and does nothing if the draft invents vocabulary your corpus lacks.
Each of the three has its own way of being wrong
- The constraint gets dropped. A rewrite that loses a date, a negation or a named party produces a fluent query for a different question, and retrieval succeeds at answering it.
- The wrong antecedent is resolved. The second one means the second contract, the rewrite decides it means the second clause, and every passage afterwards is from the wrong document.
- Expansion dilutes a precise term. Adding 6 synonyms around an exact identifier moves the query away from the only token that would have matched, which is the mechanism behind part numbers and clause references returning nothing.
- The draft answer leaks into the search. In form conversion, a model that already believes something writes a draft containing that belief, and the search then goes looking for support for it. That is confirmation with extra steps.
A rewrite nobody logged is a retrieval nobody can reproduce
Once a model sits between the user and the index, the question searched for is not the question asked, and it may differ between 2 runs of the same input. The complaint arrives as the assistant gave me a different answer this morning, and without the rewritten query there is nothing to compare — the same question returning different passages.
So log 4 things per request: the original text, the rewritten query or queries, the retrieved passage ids, and which rewrite path ran. Keeping all of that at volume gets expensive, which makes retention a sampling design rather than an on-or-off switch — the trade in head sampling versus tail sampling of AI traces. Tail sampling matters more here than almost anywhere, because the interesting rewrites are the rare ones.
Expansion also changes the load profile. One question that becomes 4 searches is 4 times the query volume, concentrated at peak hours — a capacity characteristic your store has to absorb, and one of the inputs to keeping vectors in the database you already run or standing up a dedicated store.
A rewrite stage means the system did not search for what the user asked. That is the point of it, and also the risk.
What rewriting will not do for you
Rewriting reshapes one question into one better question. It does not break a question into a sequence of dependent ones, which is a different operation with a different control flow — the structure that requires it is defined in what makes a question multi-hop.
It also cannot make an answer supported. A better query raises the odds that the right evidence reaches the model; whether the claims rest on that evidence is the separate property in groundedness as a property of an answer. Measure the 2 separately, because better retrieval and a generation step ignoring what came back look identical from outside.
What makes a rewrite stage maintainable is unglamorous: a screen showing the original question, the rewritten one and the passages each returns, for any query somebody pastes in. Without it every argument about rewriting runs on recollection. That screen is a day of work and part of the build, the way we treat it in internal tools and operations software. Related terms sit in retrieval and grounding, part of the engineering library.
Frequently asked questions
Short answers to the follow-ups this page tends to raise.
What is query rewriting in RAG?
It is a stage that transforms the user's question before the search runs, usually in one of 3 ways: resolving references so a follow-up makes sense alone, adding vocabulary the documents are more likely to use, or reshaping the question into passage-like text. The index searches the rewritten form, not the original.
What is the difference between query expansion and query rewriting?
Expansion adds terms while keeping the original question; rewriting replaces it with a different formulation. Expansion is additive and low-risk, since the original words remain in the query, while a rewrite can drop a constraint the user cared about. Both are commonly described as rewriting, which is why it is worth naming which one is running.
Does query rewriting make retrieval harder to debug?
Yes, and that is its main hidden cost. The system no longer searches for what the user typed, and a model-driven rewrite can differ between runs, so a complaint cannot be reproduced without it. Log the original, the rewrite and the retrieved passage ids together, and a class of unreproducible bug disappears.
When should I not rewrite a query?
When the query contains an exact identifier, and often when the interaction is single-turn. Part numbers, clause references and document codes are the highest-precision signal available, and paraphrasing them removes the only token that would have matched. Single-turn questions in the corpus's own vocabulary rarely need any rewriting at all.
- query rewriting
- retrieval
- query understanding
- definitions
The work behind this page
Builds from our portfolio that this page draws on.
AskVault
An AI internal knowledge-search platform that answers employee questions from your own docs — grounded in citations, with knowledge gaps surfaced and deflection tracked.
Productivity AIReply Rail
AI-drafted Google, Yelp, and Facebook review responses for local businesses — one queue, one click.
Local Business AIRead next
- The same question returns different passages every runSome of the variation is approximate search doing exactly what it was built to do. The rest is a rewrite step, a moving index or a session-dependent filter — and only that half is worth fixing.diagnostic
- Queries with a part number or a clause reference come back emptyProse questions work and a single part number returns nothing useful. Identifiers are the one query class where similarity is structurally the wrong instrument.diagnostic
- Multi-hop questions: the shape single-shot retrieval cannot serveThe test is dependency, not difficulty: if you cannot write the second search until the first answers, no amount of index tuning makes one retrieval enough.definition
- Adding one new source made unrelated answers worseNothing about the old passages changed. A new source ranks moderately well on a great many queries, and a fixed top-k has to give it those slots by taking them from something else.diagnostic
- Chunk overlap: what it protects against and what it duplicatesOverlap is insurance against a chunk boundary landing in the middle of one idea. The premium is paid in duplicated candidates crowding a fixed number of prompt slots.definition
- Facts in the middle of a long context get missedHold the passage set constant, walk the answer-bearing chunk from first position to last, and measure. The shape of the resulting curve is the diagnosis, and it takes about an hour to produce.diagnostic
Working on something in this space?
Tell us where you are in a sentence or two. We'll tell you honestly whether we're the right team, and what a sensible first slice of the work looks like.
Start the conversation