Incident, 8 September 2026 — test code wrote a false provenance value in production

Summary

For approximately one day, two facts about one business in the Intesta registry carried a verification date and a verification source that were written by our own benchmark harness rather than by an act of verification. As a result, the answer engine correctly refused to serve those facts as current — but the facts were in fact current. The visible effect was a false refusal, not a false answer.

No user data was involved. No incorrect value was ever served as true.

What happened

Our benchmark needed to test the behaviour of the system on facts that are past their freshness budget. To create that condition, the harness temporarily wrote an older verification date onto real facts and restored them afterwards, in a cleanup step attached to the end of the run.

Two things went wrong.

1. The harness set verification_source to a value that describes an act of verification, on a fact it had not verified. Test code wrote into a trust field.
2. Several benchmark runs were terminated with a hard kill, which does not run cleanup handlers. One of those terminated runs left two facts (the minimum payout and the pool fee for one coin) with the artificially aged timestamps in place.

The facts stayed in that state until they were noticed, which we estimate at around one day.

Effect

An agent or a visitor asking for those two values received refused_stale — the system declining to answer because the fact was, as far as it could tell, too old to be trusted. The underlying values were correct and current at the time.

The failure was therefore in the safe direction: the system withheld a correct answer rather than serving an incorrect one. That is by design, and it is why the effect was limited, but it is not a reason to treat the cause as minor.

How it was found

Not by us. An external review of the public API surface noticed that the verification sources for the entity had drifted from a uniform state to a split one, and that two facts were flagged stale, and asked why. The internal reports at the time said the refresh loop was healthy. It was; the harness was not.

The lesson we take from that is the more useful one: the internal report and the external check disagreed, and the external check was right.

Why this is the second occurrence, not the first

Three weeks earlier, a database migration added the verification-date column with a default of "now", which stamped every existing row with a verification time at the moment the column was created. Rows that had never been verified appeared verified. That was fixed by a follow-up migration that set the date honestly and added an explicit column recording who verified — backfill, refresher, manual or owner.

Both incidents belong to the same class: something that is not an act of verification wrote into a field that asserts verification. For a registry whose entire product is "this field can be trusted", that class is the most expensive kind of defect available.

What was changed

• The benchmark no longer connects to the database at all. Staleness is now tested by shifting the clock in the freshness computation and reading facts read-only through the public API — the data is never modified.
• The aging and restore helpers were deleted rather than made more robust. A safer version of an operation that should not exist is still an operation that should not exist.
• A rule was added to the project's standing instructions: test and benchmark code never writes to last_verified_at, verification_source, attested_at or any attestation field, and on production this is enforced by database privileges rather than by discipline. A read-only role now exists for benchmark runs.
• Staleness and expiry are computed at read time from the verification date and the category's freshness budget, rather than stored and updated by a job. A stored flag drifts; a computed one cannot.
• A related issue found in the same review — synthetic test traffic appearing in the owner-facing queue of unanswered questions — was fixed separately. Synthetic requests are now flagged, excluded from owner notifications and analytics by default, and the previously recorded rows were marked rather than deleted.

What we did not do

We did not delete the affected records, the contaminated benchmark run, or the raw output files. They are published alongside the clean results at /benchmark-rag.