The model answered from memory and ignored the passage you gave it

Build the probe first. Write a short passage that states something the model could not know and would not guess — an internal term you invented, a policy renamed to something implausible, a threshold set to an odd number — and place it in the context alongside a question only that passage answers. Run it. If the answer contradicts the passage and matches what a well-read stranger would say, you have a genuine override. If the answer is vague or generic instead, the passage almost certainly never arrived, and you have been debugging the wrong layer.

This matters because "the model ignored the context" is the conclusion teams reach fastest and confirm least. It is satisfying, it points at someone else's component, and it sends the next two weeks into prompt wording. The probe costs 10 minutes and rules it in or out.

Building the probe so its result means something

1. Write the probe passage in the register of your real documents. 2 or 3 sentences, the same tone and structure a real chunk has, so you are not accidentally testing whether the model notices an oddly formatted insert.
2. Make the claim unguessable but unambiguous. Not a fact the model might reasonably hold, and not nonsense — something like an internal escalation tier with an invented name and a stated deadline of 9 working days.
3. Ask a question whose only possible source is that passage, phrased the way a real user would phrase it, not as an instruction to quote.
4. Run it as a bare prompt first, with only that passage and the question. If the model gets it right here, the reader and the instruction are both capable, and the fault is further back in the pipeline.
5. Run it again inside the real request, with the probe passage substituted for one of the retrieved chunks so the payload length and structure are unchanged.
6. Record which answer the model gave in each condition, not whether the answer was good. The comparison is the finding; a single run tells you nothing.

Keep the probe passage in a test fixture, never in the live index. A deliberately false document sitting in a production corpus will eventually be retrieved for a real question by a real user, and the note explaining why it exists will not be attached to it when that happens.

Read the outbound request before you read the answer

Log the fully rendered request body — the exact string of messages sent, after every template, trim and formatting step — and assert programmatically that a distinctive sentence from the passage you believe you supplied appears in it as a literal substring. Not the chunk id. Not the retrieval log. The UTF-8 text, in the payload, as sent.

This one assertion catches the most common cause of the entire symptom, and it fails silently in every system that does not check it. Retrieval succeeded. The chunk was selected. Then a length-fitting step dropped it, and nothing raised an error, because trimming to fit is that step's job.

• Middle-out truncation. Several client libraries trim from the centre of a message list to preserve the beginning and the end, which is exactly where a mid-ranked passage lives.
• A per-message character cap. A chunk longer than a configured maximum — 2,000 characters is a common default — is cut mid-sentence, and the half carrying the qualifier is the half that went.
• Budget arithmetic that assumed a chunk size the corpus outgrew. If k is 12 and the budget was sized when chunks averaged 400 tokens, chunks now averaging 700 tokens mean the last 4 never fit. That trade is set out in what chunk overlap duplicates.
• A conversation history that grew until it crowded out the evidence. Nothing about the retrieval changed; the space available to it shrank turn by turn.
• A serialisation bug that renders an empty string for a chunk whose text field is null, producing JSON of the right shape and the wrong content.

The arithmetic here is worth doing explicitly rather than discovering by accident, and it is the same arithmetic that produces your bill: passages are tokens, tokens are the billing unit, and a context budget is a spend decision as much as a quality one. That equivalence is the point of the unit that makes AI spend legible.

A model cannot ignore a passage it was never sent. Most reported grounding failures are truncation failures that nobody asserted against.

The five causes, ranked by how often they turn out to be it

The last row has its own procedure, because the sweep is more informative than a single pass-or-fail: hold the set constant and walk the answer-bearing chunk through every position, as described in facts in the middle of a long context. Do that before concluding anything about attention or instruction wording.

Repairs in the order they are worth trying

1. Make the payload assertion permanent. Fail the request, loudly, when a selected chunk did not survive rendering. That is roughly 5 lines of code, and it retires an entire category of mystery.
2. Move the instruction. Put the task after the evidence, or in both positions. It costs almost nothing and it is the cheapest structural change available.
3. Cut k rather than raise it. Every passage that could not plausibly answer the question is competing for attention with the one that can, and near-identical chunks are the worst offenders because they look equally relevant.
4. Make the evidence direct. If the retrieved text only implies the answer, no amount of instruction will make the model assert it — nor should it.
5. Require a quotation. Ask for the supporting sentence before the answer, and check the quotation against the supplied text programmatically. A claim whose quotation is not in the payload is an unsupported claim, and you can detect that without a judge model.
6. Only now consider the reader. Changing the model is the most expensive move on this list and the least likely to be the fix, and it invalidates every threshold you calibrated against the old one.

Note what is absent from that list. A reranking stage changes which passages you send, not whether the model reads them, so it is the wrong instrument here — see what a reranking stage does to a candidate list for the boundary. Reranking earns its place when the right passage is in the candidate pool and ranked too low to be sent at all.

Two neighbouring failures this page is not about

If the probe passes — the passage was present, the model followed it, the override does not reproduce — then the context was used and used badly, which is a different family with different repairs. Conflation across sources, a dropped qualifier and an unsupported bridging inference all produce wrong answers from correct evidence, and they belong to the right passages and a wrong answer.

The other neighbour is what the system should do when it has no adequate passage at all. Designing a refusal that users accept, choosing the threshold that triggers it, and deciding who sees the fallback are questions about handling failure rather than about diagnosing it, and they sit outside this cluster.

The probe, the payload assertion and the quotation check together are perhaps 2 days of work, and they turn the most common argument in this whole area — is it the retrieval, the prompt or the model — into something a build log can answer. That kind of narrow, load-bearing instrumentation is what we push for early in MVP and product builds. The adjacent failure modes are collected in retrieval and grounding, within the engineering library.

• grounding
• debugging
• prompting
• context