Adding one new source made unrelated answers worse

Nothing regressed in the sense people mean when they say it. The passages that used to win still score exactly what they scored last month. What changed is who is standing above them, and because the shortlist is a fixed size, every slot the new source takes is a slot something else loses. That is why questions with no relationship to the new material get worse answers on the day it lands.

The reason this is hard to believe is that the two events look unconnected. Someone ingested a product manual set on Tuesday, and by Thursday the HR policy answers are vaguer. There is no code path linking those, and there does not need to be one: they are competing for the same 8 slots in the same ranked list.

The eviction diff, which is the only evidence worth having

Everything else in this investigation is opinion. The diff is a fact, and if you did not capture the before state you can usually reconstruct it, because most stores let you filter by source or ingest timestamp.

1. Freeze a question set of 40 to 60 real queries taken from logs, deliberately weighted towards areas the new source has nothing to do with. Invented questions defeat the purpose here — they use the vocabulary of whatever you were thinking about.
2. For each question, record the ordered chunk ids and scores of the top-k as the application actually runs it. This is the before state. If you never captured it, re-run the same queries with a filter that excludes the new ingest batch — a source column in Postgres with pgvector, a term filter in Elasticsearch, a namespace in a dedicated store — and that reconstruction is close enough to be conclusive.
3. Run the same set against the live index twice and diff the two after-runs against each other first. Approximate indexes such as HNSW can return slightly different candidates between identical runs, and if they already disagree you are looking at instability rather than eviction.
4. Diff by chunk id, not by text. For each question, list the ids that were in the before top-k and are absent from the after top-k, and note which source each replacement came from.
5. Count the evictions and attribute them. A handful of changes on questions near the new material is normal. Evictions on questions unrelated to it, all replaced by chunks from one source, is the finding.
6. Check what the replacements are. Read the text of 10 evicting chunks. In most cases they are not wrong, merely generic — which is exactly why they win everywhere and help nowhere.

That fourth row is worth separating carefully. If the winning ids are the same but the order moves between runs, the top of your list is separated by rounding rather than meaning, and the symptom belongs to the same question returning different passages every run. Ingesting anything makes that worse, but the ingest is not the cause and removing it will not fix it.

What makes a source an evictor

Sources do not evict in proportion to their size. Four characteristics do the damage, and all four describe text that is broadly plausible for many questions rather than strongly relevant to any.

The last one is the most common and the least suspected, because it looks like a settings detail rather than a ranking decision. Ingesting a source at a materially different chunk size than the rest of the corpus changes the competition for every query in the index, not just for queries about that source. If the existing corpus is chunked at 600 to 800 tokens and a new export arrives at 100 to 150, the new source is not merely present, it is structurally advantaged.

The repairs, in the order that keeps the diagnosis intact

1. Re-chunk the new source to the corpus norm and re-run the diff. If the eviction count falls by most of its value, you are finished, and this is a one-line configuration change rather than an architecture project.
2. Give retrieval a per-source budget. Fetch a wider candidate pool and cap how many slots any single source may occupy in the final list. This preserves the new material's ability to win where it is genuinely best and removes its ability to win everywhere.
3. Filter instead of ranking, where the query implies a scope. If a question is about HR policy, product manuals should not be candidates at all — the distinction between filtering and ranking is set out in metadata filters in a retrieval query.
4. Widen k and rerank the longer list, rather than widening k into the prompt. A second arm scored with BM25 helps here too, because a keyword list is ordered on different evidence and does not concede slots to a source that is merely on-topic. The trade is laid out in more candidates versus reranking.
5. Strip boilerplate at index time. Confidentiality footers and repeated headers should never have become candidates; removing them is cheap and helps every query permanently.
6. Only then reconsider the index shape. Whether the new material belongs in its own index at all is a structural question answered in one index versus per-source indexes, and it is a bigger change than the four above.

A fixed top-k is a room with a fixed number of chairs. Nobody took a chair away from the old passages; somebody else simply sat down first, on questions they had no business answering.

Three responses that make it worse

• Rolling back the ingest. It restores yesterday's numbers and teaches you nothing, and the source will be added again next quarter by someone who was not in this conversation. The arithmetic is a property of a fixed k, not of that source.
• Changing the embedding model. It re-scores everything, invalidates every threshold that depended on the old score distribution, and does not alter the fact that the shortlist has a fixed size. It is the most expensive move available and it is aimed at the wrong mechanism.
• Deleting the offending chunks by hand. This works for a fortnight and hides the structural problem, which returns with the next source that happens to be short and generic.

Three adjacent symptoms are worth ruling out first. If the queries that got worse are identifier-shaped — part numbers, clause references, codes — the diff shows absence rather than eviction: see queries with a part number or a clause reference coming back empty. If they are questions about figures in tables, those chunks were weak candidates long before the ingest, which is the number in a table the assistant cannot find. And if the new source contradicts an existing one rather than outranking it, that is a duplicate-content problem in the data layer, not a ranking contest between legitimate documents.

Make the diff a gate on ingest, not an investigation

The reason this failure survives for weeks is that answer-level evaluation cannot see it. A recall drop of 10 points at the candidate stage frequently leaves answer quality apparently flat, because the second-best passage often supports a defensible answer — right up until the question where it does not. By the time the complaint arrives, 3 more ingests have happened and nobody can attribute anything.

So run the frozen question set automatically on every ingest and compare candidate sets, not outputs. Fail the ingest when evictions on unrelated questions exceed a stated count, or when any source's slot share exceeds its corpus share by more than the multiple you have chosen. That is a small harness — a question file, a diff and a threshold — and exactly the sort of narrowly scoped internal tooling and operations software that pays for itself the first time it blocks a bad ingest.

One last symptom to connect. Where retrieval sits inside an agent loop rather than a single call, eviction does not present as a worse answer at all — it presents as a run that keeps searching, rephrasing and searching again because no result ever satisfies the step, which is one of the causes behind an agent run that never reaches a final answer. The gradual version of the same arithmetic, arriving over months instead of overnight, is retrieval degrading as the corpus grows. Both sit alongside this page in retrieval and grounding, within the wider engineering library.

• retrieval
• ingestion
• ranking
• regression