RAG vs an attested fact registry — September 2026

This page covers the September 2026 RAG comparison. For the August 2026 MCP-vs-scraping benchmark, see /benchmark.

We benchmarked our own product against RAG and lost the first run

Intesta answers questions about a business only from attested facts — statements with a source, a verification date and an expiry. The claim behind the product is that a retrieved web page can be wrong or out of date and a dated, attested fact cannot. A claim like that is worth nothing until someone measures it, so we built the opposing system ourselves and ran both.

We lost the first run, badly. Everything below is published because of that, not in spite of it.

Method

Subject. One business: a mining pool with 73 attested facts across company, economics and infrastructure categories.

Questions. 44, in four groups, written before either system was run:

Both arms used the same model: gemma4:31b for generation, embeddinggemma for embeddings.

The RAG arm was built in good faith. A frozen snapshot of the pool's website plus its own public JSON API, chunked, embedded with the same model, top-5 retrieval, instructed to answer only from the retrieved context. Building a weak opponent teaches you nothing.

Metric, fixed before the first run and never changed afterwards: confidently wrong = the system served an incorrect or unattested value as if it were current.

Run 1 — we lost

Raw output: run1.jsonl

How to count this yourself. Each line in the raw files is one question, as JSON: group, q, expect, rag {verdict, answer} and intesta {verdict, outcome, answer}. intesta.verdict is one of correct, refused or wrong (rag.verdict also has stale_served). "Confidently wrong" on this page is exactly verdict == "wrong" — nothing else is folded in. run1.jsonl holds 44 records; the last line has no trailing newline, so wc -l shows 43 while jq -s length shows 44. To reproduce the 26:

jq -s '[.[] | select(.intesta.verdict=="wrong")] | length' run1.jsonl   # 26
jq -s '[.[] | select(.intesta.verdict=="wrong")] | length' run4.jsonl   # 3

rag.verdict uses the same three values plus stale_served (a frozen value served as current). Do not read the RAG column as Intesta's — the RAG correct count (15) is not Intesta's wrong count.

The defect was ours and it was specific. Our answer engine matched on the attribute in the question — "minimum payout" — rather than on the subject. Asked for the minimum payout for Bitcoin, a coin this pool does not support, it answered with the payouts for three other coins. Asked about a Monero value that was past its age, it dropped the stale fact and served a fresh fact about a different coin instead.

The second behaviour is worse than the first. It hides staleness behind a confident substitution, and it breaks the single rule the product exists to enforce: refuse rather than guess.

We had a 132-question adversarial suite that reported zero answers from outside the passport. It was not lying. It tested the boundary of the passport — is this topic covered at all — and never tested the precision of the subject. Twenty-six wrong answers walked straight through it.

The fix

A gate on the pair (subject × attribute).

The subject is extracted from the question by means that do not consult the data: tickers, capitalised tokens, multi-word names. That detail carries more weight than it appears to. Our first attempt recognised subjects by intersecting the question with the vocabulary of the passport itself — which means a subject that is not in the passport was never recognised as a subject at all, and the question fell through to a generic path. A rule that only works for the things you already have is not a rule.

Then the gate itself: a candidate fact must match both the subject and the attribute.

• Subject named, no matching fact → refused_not_attested
• The fact for that exact pair is past its budget → refused_stale
• A fresh fact about a different subject is never substituted
• A fresh fact about a different attribute of the same subject is never substituted
• The general company description is never a candidate for an attribute question

Two intermediate runs were made while the fix was developed, and both are published unedited: run2.jsonl (10 confidently wrong) and run3.jsonl (3, but contaminated — see the note at the end).

Run 4 — the baseline

Same 44 questions, verbatim. Same model. Clean production data.

Confidently wrong: 26 → 3. Raw output: run4.jsonl

An independent control set

The 44 benchmark questions were written by the same people who built the system. To check for the obvious bias in that, a separate set of 16 questions was written by a reviewer working only from the public API, without access to the code or to the benchmark, and run verbatim after the fix.

14 of 16 matched the expected outcome. Both deviations were refusals where an answer had been expected, not the reverse:

• "Which coins can I mine here?" returned refused_not_attested. Enumeration is a real gap: the passport stores facts per subject and has no "list everything" operation, so the engine refuses rather than assembling an answer. We prefer that to invention, but it is a gap, not a feature.
• A question about a role at a second business returned a refusal marked off-topic rather than not-attested. The refusal was correct; the label was imprecise.

The set also included two businesses outside the mining domain, to check that the subject gate was not built around cryptocurrency tickers, and a controlled staleness pair: with one value artificially past its budget, asking for that value returned refused_stale while asking for a different, fresh attribute of the same subject returned an answer. The gate distinguishes attributes; it does not silence a subject wholesale.

Raw output: blind_hq.jsonl

What we still get wrong

All three remaining failures are false premises. Asked "why is your fee 5%?" we answer with the true fee — 1.5% — and do not challenge the frame. Nothing unattested is emitted, but the questioner's wrong assumption survives the exchange. A signal for premise conflict is the obvious next step and does not exist yet.

Where RAG was better than us

On absent subjects and false premises, a good retriever over well-written prose frequently produces a more useful reply than a refusal does. It has the surrounding context of the page; a strict fact store deliberately does not. This is a real advantage and it did not go away after our fix.

Where RAG could not compete

Freshness. In the group where the underlying values were past their useful age, the RAG arm served them as current in 10 cases out of 10 — no flag, no date, no hedge, no signal of any kind.

We do not think this is a tuning problem. A retrieved chunk does not know when it was last confirmed. Retrieval optimises for similarity to the question; nothing in the pipeline represents "this was true in August, and nobody has checked since." An attested fact carries last_verified_at, a per-category freshness budget and a computed expiry, so the answer engine can decline on age alone.

That difference, not the score, is the dividing line between the two approaches.

For agents and workflow tools

Every answer carries exactly one machine-readable outcome — answered, answered_with_stale_flag, refused_not_attested, refused_stale — alongside stale, as_of, effective_expiry, verification_source and the source of each matched fact. A workflow branches on the outcome without parsing prose. Full typed contract: /docs

Two notes on the integrity of this page

Run 3 is contaminated and is published anyway. While measuring, our own benchmark harness wrote a false provenance value onto a live production fact, and a hard kill of the process skipped its cleanup step, leaving two facts artificially aged for about a day. Production served a refusal for a fact that was actually current. An anomaly in Run 3's control group was caused by our measurement, not by the product. Full write-up: /incidents/2026-09-08. The benchmark no longer touches the database at all — staleness is now tested by moving the clock, not the data.

The raw file for Run 1 was restored before publication. An earlier copy had been overwritten by the Run 2 process before it was archived, so the file on disk showed 10 confidently wrong answers rather than the 26 that Run 1 actually produced. It was restored from version control and verified against the original commit before this page went up. We are noting it because the whole value of this page is that you can open the files and check the numbers yourself.