A subset of the PDFs came through as gibberish and nobody looked

Nothing in the pipeline is going to tell you. A page that extracted as a stream of replacement characters is a perfectly valid string, so it chunks, it embeds, it indexes, and it occupies a row that will never be returned for any query anyone asks. The run is green. The document count matches. The only symptom is a document that a user swears is in the system and that the assistant behaves as though it has never seen — and by the time somebody reports that, the corpus has been growing for 3 months.

This is not a rare edge. Any collection assembled over a decade contains scans without a text layer, files exported from a system that subset its fonts badly, forms with permission flags set, and pages someone photographed sideways. The proportion is usually somewhere between 2 and 15 percent, and the only way to know your number is to measure it, because none of those documents will complain.

The four numbers a legibility screen computes

Compute these per document and per page, store them alongside the extraction, and you have converted an invisible failure into a sortable column. All 4 are cheap enough to run on every file in the corpus.

1. Character-class ratios. The share of extracted characters that are letters, digits, whitespace, punctuation and everything else. Ordinary prose is dominated by letters and whitespace; a broken encoding shows a large share in the last bucket, including replacement characters and control codes.
2. Dictionary hit rate. Split on whitespace, strip punctuation, and count what proportion of tokens appear in a word list for the document's language. Clean business prose sits high even with jargon; encoding damage collapses it, because the damaged text is not words at all.
3. Empty and near-empty pages. Count pages yielding fewer than about 50 characters. A scanned page with no text layer yields close to zero, so a document that is 40 pages long and 38 near-empty pages is an image-only scan, not a short document.
4. Token-length distribution. Mean and maximum length of whitespace-delimited tokens. Encoding failures produce very long runs with no spaces; aggressive de-hyphenation produces the opposite. Both are visible as a distribution that does not look like language.

The dictionary check needs one refinement or it will flag half your corpus. Technical documents are full of part numbers, chemical names and internal acronyms that are correctly extracted and absent from any word list, so exclude tokens containing digits before computing the hit rate, and calibrate the threshold on a sample of documents you have read yourself rather than on a published figure.

Thresholds that flag for a human rather than decide alone

A screen that rejects documents on its own will throw away legitimate content — a table-heavy appendix scores badly on every one of these measures and is perfectly readable. So the threshold's job is to route to review, and the number that matters is how much review it generates.

• Set thresholds from your own corpus, not from a blog. Compute the 4 numbers across everything you already have, look at the distribution, and put the line where the tail visibly separates from the body.
• Read the flagged sample before trusting the line. Twenty flagged documents opened by hand tells you whether the screen is finding damage or finding tables, and no amount of tuning substitutes for that half hour.
• Track the flag proportion over time as its own signal. A jump from 3 to 9 percent in one week means a source changed how it exports, which is a different and more urgent problem than the historical backlog.
• Keep the scores even for documents that pass. When somebody later reports a missing answer, the first useful question is what this document scored, and that is only answerable if you stored it.

An unreadable chunk is the quietest defect in the stack. It indexes without complaint, retrieves for nothing, and leaves no line in any log.

Five ways a PDF turns into noise

The last row is worth handling separately, because it is not really an extraction problem — it is an error that a permissive parser converted into an empty string. If the extractor returns a status, persist it; if it throws, record the failure against the document rather than logging it and moving on. A pipeline that reports success while quietly producing nothing is the pattern behind most silent corpus gaps, and it is worth alerting on directly, as set out in what to alert on in an ingest pipeline.

Where the screen belongs, and why not later

Run it between extraction and chunking. Before chunking, the unit is a document and the decision is simple — index, quarantine, or send for recognition. After chunking, the damage is distributed across dozens of rows with no shared handle, and after embedding you have paid for vectors that will never be retrieved and taken on the job of deleting them cleanly.

Placing it there also gives the pipeline a natural fork. Documents that fail on empty pages go to optical recognition; documents that fail on dictionary hits with healthy character classes usually need a different extraction library rather than recognition; documents that fail on character classes alone often extract correctly on a second pass with a different tool. That fork is worth building before the corpus is large, alongside the format-normalisation work in normalising mixed formats before indexing.

One consequence to plan for: recognition is the slow, expensive path, and a backlog of quarantined scans released into it at once will saturate whatever runs it. That is the same capacity shape described in an ingest queue that backs up every Monday, and the answer is the same — meter the release rather than buying a larger machine.

What quarantine has to mean in practice

Quarantine is only useful if it is visible. A held document that nobody can see is indistinguishable from a lost one, and the whole point of the screen was to stop losing documents silently.

• A queue somebody owns, with the scores, the first page rendered and the extracted text side by side. Deciding takes seconds when you can see both; it is impossible from a score alone.
• Three outcomes per document: re-extract with a named tool, send for recognition, or accept as-is with a reason. The third exists because some documents genuinely are tables of numbers.
• A count on a dashboard, not a log line. The number of quarantined documents and the age of the oldest are the 2 figures that tell you whether the process is working.
• An answer for users. If a document is held, a search for it should say so rather than returning nothing, because a user who is told the file is being processed does not file a bug.

That queue is a small, unglamorous piece of software, and the decision about whether it deserves to exist as a real tool rather than a shared spreadsheet is a recurring one — the case is argued in whether a small internal tool is worth owning. It is also the kind of thing worth including in a first release rather than deferring, which is how we scope product and MVP builds that depend on a document corpus.

Quarantine also has a cost that has to be declared. A document held for review is a document not answerable, and if anyone has promised how current the corpus will be, the holding time counts against that promise. Work out where it lands against the commitment described in what a freshness commitment actually binds you to before the queue grows.

What a clean score still does not prove

The screen measures whether the characters are language. It says nothing about whether the language is in the right order, and that is the failure it structurally cannot see. A 2-column page read straight across produces real words, real sentences fragments and a perfect dictionary hit rate, while the actual meaning is shredded — the first half of every line from the left column glued to the first half of every line from the right. Catching that needs a layout check or a human reading a sample, and a sample is cheaper.

Two further things it does not settle. Re-extracting the corpus after fixing a parser is a change the pipeline will probably not notice, because the source file is untouched and change detection is keyed to the file — the same blind spot described in an updated document that still serves old text, and the reason an extractor version belongs in whatever signals a re-index. And a corpus can be perfectly legible and still contradict itself, which is two copies of one document and an assistant that believes both. Both sit in data readiness and pipelines, part of the engineering library.

• document extraction
• ingest quality
• ocr
• corpus hygiene