Metadata filters, and the difference between filtering and ranking

A metadata filter is a rule over the structured fields stored with each chunk — source, date, department, document type, tenant, permission key — deciding which passages may be returned at all. It is binary: a passage that fails it is not ranked low, it is not in the running.

That is the whole distinction from a boost, and worth being pedantic about: the 2 are described with the same words in most documentation and fail in opposite directions. A filter set too tight returns nothing. A boost set too weak returns everything, in a slightly better order.

Admissibility is binary, preference is continuous

The last row settles most arguments. A filter is a statement someone can read: this query considered only documents this user may open. A boost is a number whose behaviour shifts with every list, so it underwrites nothing.

The fields worth carrying on every chunk

Filters can only use fields that exist at query time, so what to store is decided at ingest, long before anyone knows which filter they need. Retrofitting one means a reindex.

• Tenant and permission key. The 2 fields that must never be a boost, and worth testing on every release.
• Source system and native record id. Lets you exclude a whole connector after a bad import, and a citation resolve to something a person can open.
• Document type and status. Draft, superseded, approved, template — the most useful filter in most corpora, and usually the last one added.
• Effective and expiry dates, distinct from the ingest date. A policy live from April and replaced in September needs both, and neither is the day it was indexed.
• Content hash and pipeline run id. Operational rather than user-facing, and the difference between isolating a bad batch and rebuilding all of it.

Filtering before the search and after it are different algorithms

Post-filtering runs the search first and discards candidates that fail the rule. It is trivial on any store, and it silently shrinks the result set: the search had no idea the filter existed when it chose those candidates.

Pre-filtering restricts the space before or during the search, so every candidate is admissible. It needs index support, and the cost lands on the engine: against a very selective condition an approximate index may widen its search or fall back to scanning, so latency becomes a function of how much of the corpus the filter admits. That data-dependent tail is what an objective must be written against, not an average — the distinction in SLA versus SLO for an AI feature.

Between them sits over-fetching: ask for more candidates than you need, then post-filter, accepting that the multiplier is a guess. It holds while filters are loose and breaks when someone adds a narrow one — recall collapsing the day permission filtering was turned on.

A boost demotes a passage the user may not see. A filter means it was never a candidate. Only one is a permission model.

Why a filter mistake is quieter than a ranking one

A bad ranking is visible: the answer is weaker, someone complains, you look at the order. A bad filter destroys the evidence of its own mistake. The passage that would have answered is absent rather than low, and the model composes something confident from what survived.

• An overly tight condition returns 2 passages instead of 10, so the answer reads thin rather than wrong.
• A filter value derived from the question can be extracted wrongly — last year becoming the wrong date range — a failure of query rewriting before retrieval, not of the filter.
• An exact-match predicate over free text is brittle in the way lexical and semantic retrieval fail differently describes: Legal and legal-department differ.
• A filter applied to only one arm of a hybrid search leaks through the other, which is how permission bugs survive code review.

Because filters decide which evidence is admissible, they decide what an answer can be supported by. An assistant that filtered out the governing document and answered anyway produced something confident and unsupported — groundedness as a property of an answer. Log the filter set beside the query, as part of that answer's provenance.

Choosing which metadata to capture is a schema decision taken in the first version, when it is cheap, rather than after the first reindex, when it is not — which is why we treat retrieval metadata as product design in MVP and product builds. Where the permission check sits is adding permission filters to retrieval. Neighbouring terms are in retrieval and grounding, part of the engineering library.

• metadata
• filtering
• vector search
• definitions