The reindex has been running for three days and is still not done

Nobody wants to kill a job that might be 90 percent done. That is exactly why a stalled reindex survives for 3 days: the progress bar moves, the process is alive, and nothing in the logs says failure. The first question is not how to speed it up, but whether it is progressing at all.

Percent complete is the wrong instrument because it moves for reasons unrelated to work done. The denominator is often the total row count rather than the records that actually need processing, retried records are counted more than once, and a restart resets the numerator without resetting anyone's expectations. Rate does not have those problems.

Plot committed records per minute across the whole run

Committed means durably written and visible to a subsequent read, not read from the queue and not sent to the embedding endpoint. Anything short of committed can be lost by a restart, which is precisely the case you are trying to detect.

1. Emit a running committed count with a timestamp every 30 seconds, or reconstruct one by counting rows in the target index bucketed by their write time.
2. Convert to a rate in 5-minute buckets and plot the whole run, not the last hour. The shape over hours is the diagnosis; the instantaneous number tells you nothing.
3. Estimate the finish from the last hour's rate, then again from the first hour. If the 2 estimates differ by more than a factor of 2, the rate is decaying and the average is lying to you.
4. Count how many records actually need work — changed since the last successful run — and use that as the denominator instead of the table size.
5. Check whether the committed count ever goes backwards or plateaus at a round number. Both mean a restart, and a restart without a checkpoint means the run keeps reprocessing the same opening slice.

Throttling that a retry loop turns into patience

The failure that hides best is the one your code already handles. A hosted embedding endpoint answers a request beyond its limit with HTTP 429 Too Many Requests — the mechanism MDN describes as asking a client to slow down — often with a Retry-After header saying how long to wait. A client that sleeps, retries and never gives up converts a hard limit into an indefinite wait with no error anywhere in your logs.

• Count retry attempts as a metric, not a log line. Attempts per minute plotted next to commits per minute makes throttling visible at a glance.
• Count seconds spent sleeping. A run where 80 percent of wall-clock time is backoff is not compute-bound, and adding workers makes it worse.
• Honour Retry-After rather than doubling blindly. A client backing off harder than the server asked wastes the quota it was just granted.
• Bound the retries. After a fixed number of attempts a record belongs in a dead-letter store with its error, so the run can finish and the failures can be counted.
• Cap concurrency below the published limit. Throughput against a throttle is usually higher at 8 steady workers than at 32 all being refused.

Where the limit belongs to a vendor, the ceiling is theirs to move rather than yours. Quotas, batch maximums and per-minute allowances change on somebody else's schedule — one of the dependencies weighed in whose roadmap you inherit when you buy. Design the run so a lower limit means a slower finish, not a failed one.

A job that logs nothing is not necessarily working. Silence in the log is not silence on the wire.

Containment for a run that is already in flight

You usually have to decide something before the diagnosis is complete, and the order matters. Do not kill the process first: a running job holds state that is often the only record of what has been done.

1. Capture the state before touching anything: highest committed key, committed count, retry counters. If none exist, that is the finding, and the run cannot be resumed whatever you do next.
2. Protect live traffic first. If the rebuild writes into the index being served, stop those writes and let the old content be served stale.
3. Shrink the work rather than speeding it up. Filter to records whose content hash changed, and to the sources people actually query.
4. Lower concurrency if retries are high. Counterintuitive and usually right, because a throttled endpoint rewards a steady rate over a burst.
5. Restart with a checkpoint even mid-incident. Recording the highest completed key every few thousand records turns one 3-day gamble into a series of short reversible ones.
6. Set an abandon threshold before you walk away. If the rate does not double within an hour of the change, stop the run rather than losing another night.

Most of a full rebuild is work that did not need doing

Between 2 rebuilds of a document corpus, the share of content that genuinely changed is often in the low single digits. Re-embedding all of it treats a small delta as a full corpus, and that is the difference between a run that finishes overnight and one that does not finish at all.

• Hash the normalised chunk text and store it. Skip any chunk whose hash matches what is indexed, and the run collapses to the real delta.
• Hash after normalisation, not before. Whitespace, extraction quirks and a re-exported PDF otherwise present unchanged text as new work every run.
• Keep the embedding model identifier beside the hash. A model change genuinely invalidates everything, and that is the one case where a full rebuild is correct rather than lazy.
• Build into a second index and swap the alias once verified. A rebuild mutating the index that serves live traffic is an outage with a schedule attached.

Whether the norm should be incremental with periodic full rebuilds is a standing decision rather than an incident response, argued out in rebuilding the whole index or updating only what changed. It gets sharper the faster the corpus churns: a document set that turns over daily, like dispatch notes and proofs of delivery, makes it operational rather than architectural, which is the texture described in AI in logistics operations.

Making the next run interruptible by design

The property that matters is not speed. It is that stopping the run at any moment costs you at most one batch, so a rebuild becomes something you can pause during business hours and resume overnight.

• Checkpoint on a stable, ordered key that survives re-export and rename — the same key choice that decides whether idempotent ingestion is achievable at all.
• Make writes idempotent. Reprocessing a checkpointed batch after a crash must overwrite rather than duplicate, or every restart inflates the index.
• Carry a run identifier on every written record, so a partial run can be identified, measured and rolled back as a unit.
• Emit progress as a rate and alert when it falls below a floor for a stated window — the absence-shaped alerting in what to alert on in an ingest pipeline.
• Rehearse an interruption. Kill the job deliberately in staging and confirm it resumes. A resume path never executed is a hypothesis.

None of this is model work, which is why it is left until the night it matters. Checkpointing, retry policy, dead-letter handling and an alias swap are the ordinary orchestration around an AI system, and we treat that plumbing as part of the build in AI agents and automation. The staged approach for a rebuild that must run against a live system is running a backfill without taking the system down.

What finishing the run will not fix

A completed reindex guarantees that everything in the corpus is now represented in the index. It guarantees nothing about whether that content should have been indexed, and a rebuild is the most expensive way to propagate a quality problem across every record at once.

• Unreadable extractions embed happily and retrieve for nothing, so screen them before the run using the legibility checks in PDFs that came through as gibberish.
• Duplicate entities are re-embedded as faithfully as anything else, and the rebuilt index reproduces the split totals in one customer under three spellings.
• A field that quietly went empty upstream leaves the same value missing everywhere, which a rebuild spreads rather than repairs — found by the comparison in an upstream field that changed while the pipeline carried on.

Run the quality screens first, then the rebuild. Reversing that order costs the same days twice, and the second run is always the one under scrutiny. All of it sits in data readiness and pipelines, part of the engineering library.

• reindexing
• throughput
• rate limits
• diagnosis