Queries with a part number or a clause reference come back empty

Identifiers are the one query class where embedding similarity is structurally the wrong instrument, and nothing in the retrieval stack will fix it: not a better model, not a larger k, not a reranker. An embedding compresses meaning into a fixed-length vector. A part number has no meaning to compress. What the vector ends up encoding is the words that happened to surround the code in the chunk, which is why a search for one code cheerfully returns passages about a different one.

Prose queries keep working throughout, which makes this confusing to report. 'What is the warranty period on the pressure regulator' answers fine. 'What is the warranty on ABC-4471-B' returns three paragraphs about a different product line, and the user concludes the system is broken generally.

Confirm it with an exact substring scan

Before any theory, establish whether the string exists in the corpus at all. Ten minutes, and it splits the problem cleanly in two.

1. Take 20 identifiers that users have actually searched for and that you know are documented. Pull them from query logs, not from a catalogue.
2. Scan the raw source files for each one as a literal substring — grep over the extracted text, or a LIKE query against the text column you indexed. Record present or absent.
3. For each identifier present in the raw text, scan the chunk table for the same string. A code present in the file and absent from every chunk means it was destroyed between extraction and indexing.
4. For each identifier present in a chunk, run it as a query through the live retrieval path and record whether that chunk appears in the top-k at all.
5. Tally the three columns. Where the identifier survives into a chunk and still does not retrieve, you have confirmed a scoring problem rather than a pipeline problem, and the repairs in this page apply.

The second row sends you somewhere else entirely. Identifiers very often live in exactly the structures that parsing damages — a specification table, a column header, a repeated page header — which is the failure described in the number in a table the assistant cannot find. No amount of retrieval work helps when the string is not in the index.

Why similarity cannot rank an identifier

Four mechanisms, and they compound. Understanding them stops a team spending a quarter on embedding experiments.

• Subword tokenisation shreds the code. Embedding models split rare strings into fragments, so ABC-4471-B becomes a handful of pieces that thousands of other codes also contain. The model never sees the identifier as a unit, and the fragments carry no distinguishing signal.
• A one-character difference is semantically invisible. ABC-4471-B and ABC-4471-C are, to a similarity function, essentially the same point in space. For a reader they are different products with different fittings, and returning the wrong one is worse than returning nothing.
• The query is too short to carry context. A 3-token query gives the encoder almost nothing to work with, so the resulting vector sits in a low-information region and matches whatever is generically nearby — often the most verbose chunk in that neighbourhood.
• Rare strings have unstable representations. A code that appeared a handful of times in training, or never, gets a vector assembled from fragments rather than learned as a concept, and small formatting differences move it unpredictably.

Similarity search answers 'what is this like'. An identifier query asks 'where is exactly this', and those two questions do not have the same instrument.

The four causes worth checking, in order

The second row is the quiet one. A document set written over 10 years will hold the same code as ABC-4471-B, ABC 4471 B, abc4471b and, in one system's export, ABC-04471-B with a zero-padded middle segment. Each is a different string to a keyword index and a slightly different point to an embedding, so a repair that normalises only the query fixes about half the corpus and reads as an intermittent bug afterwards.

The two repairs, and the test that picks one

Both are ordinary engineering. Choosing between them is a property of your identifiers, not a preference.

The deciding test takes an hour. Sample 100 identifiers from real queries and try to write a regex that captures them. If a single expression captures 95 percent or more, extract the field: an exact filter on metadata is faster, more precise and easier to explain than any ranking change. If your sample needs 4 expressions and still misses a tail, build the lexical arm instead, because you will not keep 4 patterns current. The argument for running both arms permanently is made in keyword matching versus vector similarity, and merging the two ordered lists is its own decision, covered in rank fusion across two result lists.

One deployment caution on the extracted-field route. An exact filter is a filter, and where it is applied decides whether it helps or hurts: applied inside the search it narrows the space before ranking, applied after a nearest-neighbour search it eats a candidate set that was already fixed at k. That is the same arithmetic that produces recall collapsing the day permission filtering was turned on, and it catches identifier filters just as reliably.

What the answer layer must do with an identifier

Retrieval is only half of it. An identifier is exactly the kind of token a language model reproduces approximately, and an approximate part number is worse than no answer: it looks authoritative and orders the wrong component.

• Require verbatim provenance. Any identifier in the output must appear character for character in a retrieved passage, and this is cheap to check programmatically after generation rather than hoping the prompt held.
• Refuse rather than approximate. If the identifier in the question does not appear in any retrieved passage, the correct output says so. A model asked about a code it never received will supply a plausible one from memory, which is the failure pattern in the model answering from memory instead of the passage.
• Echo the user's identifier back in the answer. It gives the reader a one-glance check that the system understood the code it was asked about, and it makes transposition errors obvious.
• Never re-derive a code from a description. If the passage says the regulator, and the user asked about a specific part number, the answer must not connect them unless the passage does.

One cost detail before you wire the lookup in. If an identifier lookup injects a record into the front of the prompt, it changes the prefix on every request and destroys any cached-prefix saving you relied on. Put stable instructions first and the resolved record after them, for the reason set out in a prefix cache that stopped paying for itself.

Clause references are the harder version of the same problem

A part number is at least globally unique. A clause reference is not: nearly every contract in a set has a section numbered 12.4, and the correct one depends on which agreement is meant. Treat the reference as a search string and you get the most verbose clause 12.4 in the corpus, from a document nobody asked about.

So a clause reference is a two-part key — document identity plus reference — and must be resolved as one. Extract the section number to metadata at ingest, filter by document first and reference second, and if the question names no document, ask rather than guess. Where the clause is the natural unit of retrieval, those boundary decisions are covered in choosing chunk boundaries for contracts.

None of this is exotic work — an extraction rule, a second index or a filter, and a post-generation check — but it sits outside what a general retrieval library gives you by default, and it is usually discovered late. It is the kind of bounded build we scope under AI agents and automation, and who should build it is the subject of choosing an AI development partner. This page sits with retrieval and grounding, inside the wider engineering library.

• retrieval
• lexical search
• identifiers
• tokenisation