Lineage: tracing one answer back to the record and the run that produced it

Lineage is the ability to name, for any passage a system served, the source record it came from, the version of the transform that produced it, and the pipeline run that wrote it. Three identifiers, carried on the derived record itself. Everything people want from lineage — explaining an answer, scoping a repair, reproducing a result — is a query over those three fields.

The three identifiers, and what each one buys

The first is the one everyone builds. The second appears the first time an extractor is upgraded and somebody asks which passages predate the change. The third is almost never designed in, because on the day you build the pipeline there is no question it answers.

Why the run identifier is the field people regret

A run identifier costs almost nothing. Generate one per execution, write it on every record that execution touches, and index it. A 26-character sortable identifier on 5 million chunks is a few hundred megabytes at most, and it turns the worst question in an incident into a single query: select everything this run wrote.

Reconstructing it afterwards is a different order of work. Job logs have a retention period measured in weeks — 30 or 90 days in most default configurations — while derived records live for years, so by the time a bad passage is noticed the log that explained it is gone. What remains is a timestamp on the row.

The two questions you ask the morning after a bad answer

Incidents in a retrieval or generation system reduce to these, and they get asked in this order.

1. Where did this text come from, and is the source actually wrong? The source record and version answer it in one lookup. Often the source is correct and the transform mangled it — a different repair, and a faster one.
2. What else is affected? Select every derived record carrying the same run identifier and transform version. That set is the repair scope. "14,000 chunks from Tuesday's run" is a reprocess with a known size; a guess is an argument.
3. Would a re-run produce the same text? Compare the transform version on the bad record with the current one. Where they differ, reprocessing fixes it; where they match, the fault is in the transform and reprocessing reproduces it faithfully.

Two properties elsewhere decide whether those queries return anything useful. The record key has to survive a replay, or the second run of the same file creates new rows and the lineage points at a record nobody serves — the reason running the same file twice must change nothing. In a streaming design "a run" has to become a checkpoint or offset range, because there is no job boundary to attach an identifier to, which is an underrated cost in scheduled batches against a stream.

What lineage is not, however often the words get swapped

• It is not a citation. A citation is what a reader sees, a link to a document and a passage. Lineage is what the system stores, including things no reader wants, such as which run wrote the row.
• It is not an audit trail. An audit trail records human actions: who approved, who accessed, who overrode. Lineage records machine derivation, and the two usually live in different places.
• It is not a merge decision. When 2 sources disagree, lineage records that both existed and which was written; the rule deciding the winner is the golden record in a messy source set.

The practical argument for building it early is not compliance. A pipeline without lineage can only be debugged by whoever remembers how it behaved, and that person leaves — the failure mode weighed up in who owns the outcome when the work is outsourced. Three identifiers per derived record is what makes a system reasonable to a new engineer, and a default worth insisting on in AI automation and agent engineering, for the reasons in what an AI-native product studio actually is. The rest sits in data readiness and pipelines, part of the engineering library.

• lineage
• provenance
• definitions
• pipelines