Someone updated the document and the system still serves the old text

The policy was corrected on Tuesday, the ingestion job has run 14 times since, every run reports success, and the assistant still quotes the old clause. The job is almost certainly fine. It asked the source what had changed, the source said nothing had, and it correctly did nothing. The defect is in the question, not in the answer.

The distinction matters because it points at a different half of the system. Debugging the indexer means reading job logs that keep saying the same true thing. Debugging the signal means making an edit you control and watching what each mechanism reports about it, which takes about an hour and ends in a definite answer.

Edit a canary document and watch each signal in turn

Put a document you own into every source the pipeline reads, and make four deliberately different edits to it, recording the raw signal values before and after each one. Do this in the real source system, through the interface real users edit with: an edit made through an API frequently updates metadata that the same edit through a web UI does not, and the reverse is also common.

1. Record the baseline. Capture the source's modification timestamp to full precision, any version or revision counter, any entity tag or checksum the source exposes, the byte size, and the hash your pipeline actually computes. Six values, written down, before anything is touched.
2. Change one word of body text and re-record all six. This is the edit that must be detectable. If nothing but the byte size moved, you already know the timestamp is not a usable signal in this source.
3. Change only metadata — a title, a tag, an owner — and re-record. A pipeline that reprocesses on this wastes work; one that ignores it misses the day someone corrects a document title people search by.
4. Move or rename the file, then re-record. Path changes are the commonest cause of one document appearing twice under two identities, which is why the ingestion key should not be the path — see running the same file twice and changing nothing.
5. Have the edit made by an automated account, a bulk tool or a restore from backup, if any of those touch this corpus. File synchronisation tools and restores preserve the original modification time on purpose — it is an option and it is on in most default presets — and that is the single most common way a real edit becomes invisible.
6. Run the pipeline and record which of the 4 edits it detected. An edit that moved a value but produced no reprocessing is a mapping problem in your own code, not a source problem, and a much faster fix.

Six ways a real edit produces no signal

Each row has an observable on the canary. Match what you recorded rather than reasoning about which is most likely: 2 of these are invisible to inspection and appear only in a before-and-after comparison.

The fourth and fifth rows are the same decision made in opposite directions, and neither is safe alone. The OOXML office formats — .docx, .xlsx, .pptx — are ZIP containers, so rewriting one with identical content produces different bytes and a container hash reports a change that is not there. Hash the extracted text instead and the opposite failure appears: a scanned page where the correction lives inside the image and only OCR would see it, a value the parser drops, a footer the extractor never reads. Keep both. A SHA-256 digest is 64 hex characters, so carrying one over the raw bytes and one over the normalised text costs 128 characters per record and closes both blind spots.

The entity tag you are treating as a checksum

Object stores and web servers expose an entity tag on every object, and pipelines lean on it as a content fingerprint. Read the documentation before you do. Amazon's S3 API reference defines its ETag as an opaque identifier for a specific version of a resource at a URL — it makes no promise about how the value is derived, so any equality logic built on it is your assumption rather than the store's contract.

The stronger checksum headers the same API exposes — CRC-32, SHA-1, SHA-256 and others — carry 2 documented caveats. They are present only if a checksum was supplied at upload, so a bucket loaded by 3 tools over 5 years has them on some objects and not others. And for a multipart upload the value may be a calculation over the checksums of the individual parts rather than a digest of the whole object, so the same bytes uploaded with a different part size can carry a different value. Check the store you use, on objects uploaded the way yours are.

The job did not fail to see the change. It asked a question that could not have returned one, and then reported success for doing exactly what it was told.

When the source and the index agree and the answer is still old

If the canary proves detection works, the stale text lives somewhere after the index. Query each store directly with a distinctive phrase from the new version and stop at the first that returns the old wording: that store missed the update, and everything upstream of it is exonerated.

• The retrieval index. Search it for the new phrase. Present in the index and absent from the answer means retrieval rather than ingestion, and the problem moves to ranking.
• The vector store, when it is a separate system. Text updated in one and embeddings replaced in the other leaves 2 representations of a single document that disagree.
• Derived artefacts. Summaries, extracted fields and generated titles are computed once and rarely recomputed on update, so they quote the old text long after the passage itself is correct.
• The answer cache. A cached response keyed on question text serves the old answer to the identical question for the whole of its lifetime, however current the index is.
• Anything in front of the source. A content delivery cache or proxy can serve a stale copy to the pipeline itself, in which case the pipeline never saw the new bytes at all.

Whichever store held the old copy, the durable answer is that an update must be a first-class operation with a verification query per store, exactly as a deletion is — the design in making a delete travel all the way to the last cache. Wiring that fan-out into an assistant's refresh path, with a check after each hop, is ordinary AI automation and agent engineering.

How much of this you get to choose

Some of these repairs are not available to you. A managed connector platform picks its own change-detection strategy, usually timestamp polling, and rarely documents the blind spot: you get a working connector and someone else's assumption about what counts as a change. A connector you host lets you hash what you like and reconcile when you want, and costs engineering time nobody budgeted. That is the general trade-off in buying a commercial platform against hosting the open-source equivalent.

The same question decides how much of the chain you can inspect. Where the stack runs on infrastructure you control, every cache between the source and the answer is yours to flush and yours to instrument — an underrated argument in running a private language model deployment. Where it does not, a periodic reconciliation sweep is the only tool that reliably repairs a missed change, whatever caused it.

What a working signal still leaves undone

Detection tells you something changed. It does not say how quickly the change must reach a reader, and a pipeline that notices an edit within seconds but runs nightly is a day stale by design. That target is a commitment somebody has to make and measure — the point of what a freshness commitment obliges you to build.

Nor does a repaired signal say which mechanism you should have chosen, or what the replacement is blind to. Every option trades one blind spot for another, and choosing deliberately is change data capture applied to a document corpus, in data readiness and pipelines within the engineering library.

• change detection
• ingestion
• staleness
• diagnosis