Intesta · evidence

Benchmark: agent over MCP vs website scraping

Run on 18 August 2026 · acceptance criterion of Phase 1 · the first public proof of value. Translated from the internal record (BENCHMARK.md in the repository); numbers are unchanged.

Method

Task: 10 identical control questions about the mining pool smoozypool.es (fee, payout scheme, minimum payouts, algorithms, stratum ports for specific coins).
Arm A (MCP): a Claude agent whose only source is https://intesta.io/mcp (tools ask / get_passport). The pool website is forbidden.
Arm B (scraping): the same class of agent whose only source is the HTML/JS pages of smoozypool.es. Its /api/* and other sites are forbidden.
Judge: a third agent, strict comparison against ground truth captured from the live /api/pools before the start. Verdicts: correct / wrong / not_found.
Both arms ran in parallel and independently; network time was measured around every request.

Results

Metric	MCP (Intesta)	Website scraping
Correct answers	9/10	9/10
Wrong answers	0	1 (ZEPH minimum payout: "1.0" instead of 5.0)
Honest "not found"	1 (Tari)	0
HTTP requests	2 (get_passport + one control ask)	3
Total network time	0.93 s	1.51 s
Full agent wall time	seconds (not fully instrumented)	93.7 s (HTML/JS parsing)

Two main findings (both re-checked by hand after the run)

The scraper lied, MCP did not. The pool website (/coin-config.js) still contains an outdated minPayout: '1.0 ZEPH', while the live API returns 5.0. The scraping agent confidently returned the wrong value; the MCP agent answered correctly from the passport. This is exactly the failure Intesta protects against: a confident wrong answer is worse than an honest refusal.
The passport was fresher than the "fresh" ground truth. Tari disappeared from /api/pools during the day; the passport (daily refresh plus fact revocation with ledger history) reflected it and the MCP agent honestly answered "not found", whereas the ground truth captured an hour before the benchmark still listed Tari. The judge scored this as not_found for the MCP arm; in fact the behaviour was more correct than the reference.

Limitations (stated plainly)

The MCP arm returns verbatim passport facts (no LLM rephrasing was connected); factual accuracy was scored, not style.
Full wall time of the MCP arm was not instrumented (the agent measured network only); the comparison 0.93 s of network vs 93.7 s of scraper work partly reflects parsing cost, not only transport.
Both arms were the same agent class (Claude) with the same task; a run with third-party agents from other vendors is the next iteration.
The entity's passport was collected from the same public pool API as the ground truth — agreement on 9 questions is expected; the value shows on the discrepancies (ZEPH, Tari).

Reproduction

Workflow script intesta-benchmark (run log in the 2026-08-18 session). Ground truth: a snapshot of /api/pools. Both arms' answers and the judge's verdicts are in the workflow result (JSON); the key rows are reproduced above. Try the same questions yourself: smoozypool.es passport or the MCP endpoint https://intesta.io/mcp.