Recall collapsed the day you turned on permission filtering

Nothing about retrieval quality changed. A filter applied after the nearest-neighbour search cannot add anything to the candidate list — it can only take things out of a list whose size was decided before it ran. If the search returns 20 candidates and a user can see 8 percent of the corpus, the expected number of survivors is between 1 and 2. That is the whole failure, and it is arithmetic rather than a tuning problem.

It presents unevenly, which is why it gets misdiagnosed. Administrators and long-tenured staff see everything and report no problem at all. Contractors, new joiners and anyone in a narrow function get vague answers or none. Aggregate quality metrics average those together and show a mild dip, so the investigation starts in the wrong place.

Measure candidate survival per role, not per query

One measurement settles this before any redesign. It compares the candidate set with the filter on and off, for the same queries, segmented by the thing that actually varies: who is asking.

1. Pick 30 real queries from logs and 5 representative principals — an administrator, a full-time member of a large team, someone in a narrow function, a contractor, and a new joiner in their first week.
2. For each query and principal, run retrieval twice: once as the application runs it, and once with the access filter removed. Record the candidate count and the ordered chunk ids for both.
3. Compute survival: filtered candidates divided by unfiltered candidates, averaged per principal. This single ratio is the diagnosis.
4. Compute each principal's visible share of the corpus — the count of chunks they may see, divided by the total. This is the number the survival ratio should be compared against.
5. List the queries where survival is zero. Those are the users currently receiving answers assembled from nothing, and they should be the first thing anyone looks at.

Four causes, ranked

The fourth cause is worth dwelling on because it is the most defensible decision with the worst side effect. Defaulting to deny is correct: a document whose permissions are unknown should not be served. But in a corpus assembled from several systems, the material with no access list is usually the oldest and most authoritative — the policies, the standards, the original contracts — because it predates whatever permission model was later adopted. Count those chunks. In most corpora the number is larger than anyone expects.

The third cause has a particular signature worth recognising. Membership expanded at ingest and frozen into each chunk means the index encodes an org chart from the day it was built. Someone changes team, keeps access they should have lost or loses access they should have kept, and nothing anywhere reports it. The permission model is not wrong; it is stale, which is harder to see and easier to leave.

A post-filter can only remove. If the list it receives was fixed at 20 and the user may see one in twelve, the answer was decided before the filter ever ran.

Three places a filter can sit, and what each costs

The middle row carries a latency caveat that surprises teams who move the filter inwards and expect a pure improvement. A very selective filter inside an approximate index can push the search towards scanning far more of the graph than usual, so the median stays flat while the tail stretches. Where the service also scales to zero between requests, that tail lands on top of a cold start and turns into a timeout — the compound failure described in the first request after a quiet hour timing out. Measure p95 per role after the change, not overall.

What to fix, in order

1. Audit access metadata coverage. Count chunks with no access field, and count chunks whose access field disagrees with the source system today. Fixing the query path over bad metadata produces a system that is confidently wrong instead of visibly broken.
2. Resolve identity at query time. The request should carry a principal, and group expansion should happen against the identity provider on that request, not be baked into the index.
3. Move the filter inside the search where the index supports it, and measure both survival and p95 per role before and after.
4. Where it cannot move, over-fetch by the multiplier for that principal rather than a fixed number. A single global k cannot serve an administrator and a contractor at the same time.
5. Add a floor alert. If a principal's candidate set after filtering is below a stated minimum, log it as an event. Silent starvation is the reason this survives for months.
6. Re-run the survival measurement on a schedule. Access changes constantly and this is the only metric that notices.

What a starved candidate set actually does to the answer

It does not produce an empty response, which is what makes this dangerous. Handed one marginal passage, a model will usually still answer, filling the gap from its own parameters and adopting the confident register of the passage it did receive. That is the failure pattern in the model answering from memory instead of the passage, and permission starvation is one of its most reliable causes.

There is a second, subtler effect. When only one or two candidates survive an approximate search, small run-to-run variation in the candidate set changes the entire answer rather than reordering it, so the same person asking the same question twice gets materially different responses — the instability described in the same question returning different passages every run. Users report that as unreliability, not as an access problem, and the ticket goes to the wrong team.

One boundary to be explicit about: this page is about lost recall, not exposure. Whether a filter can be bypassed, what happens to a passage after a document is deleted, and how tenants are kept apart are separate questions with separate failure modes, and they live in security, privacy and access. If your survival measurement shows the filter having no effect at all, stop reading this page — that is an exposure finding and it is more urgent than recall.

Retrofitting permissions onto a pipeline that never had them is its own piece of work, sequenced in adding document-level permissions to a retrieval pipeline. The measurement harness described here — survival per role, run on a schedule, alerting on a floor — is a small internal tool rather than a project, and it is the sort of thing we scope under internal tooling and operations software. It sits with the rest of retrieval and grounding, inside the wider engineering library.

• retrieval
• access control
• recall
• filtering