Evaluation v1.0 in full: the 36-task recovery ablation, drawn and downloadable.
Evaluation v1.0 — 36-task suite
Version label: Evaluation v1.0 — 36-task suite, recovery ablation. The same agent runs the identical 36 tasks twice in headless Chromium (Playwright): once with the recovery half of its loop on, once with it off (no re-grounding, no overlay dismissal, no retries — the first failure ends the task). Every step carries an explicit postcondition written against the page; a task passes only when every postcondition holds. Retries are budgeted at most 2 per step. The suite is 27 control tasks and 9 adversarial tasks (the page mutates or obstructs mid-task), in six categories: Forms & wizards (10) · Tables, lists & cart (6) · Adversarial recovery (9) · Widgets & dialogs (5) · Business workflows (3) · Portal navigation (3). Method in full: Evaluation method.
Results
36/36Tasks passed with recovery on (the citation gate) · 95% CI 90.3–100%
27/36Tasks passed with recovery off · 95% CI 57.8–87.9%
9/9 vs 0/9Adversarial tasks passed, recovery on vs off · 95% CI 66.4–100% vs 0–33.6%
27/27Control tasks passed both ways
Category
Tasks
Recovery on
Recovery off
Forms & wizards
10
10/10 (100.0%)
10/10 (100.0%)
Tables, lists & cart
6
6/6 (100.0%)
6/6 (100.0%)
Adversarial recovery
9
9/9 (100.0%)
0/9 (0.0%)
Widgets & dialogs
5
5/5 (100.0%)
5/5 (100.0%)
Business workflows
3
3/3 (100.0%)
3/3 (100.0%)
Portal navigation
3
3/3 (100.0%)
3/3 (100.0%)
Pass rate by category — recovery on vs off
Every task, both modes
Total attempts: 185 with recovery on, 171 off — identical on the control tasks (148 each), so every extra attempt sits in the adversarial group, where 11 recoveries were spent. With recovery on there are no failures to list; with it off, the nine failures are recorded verbatim on the web-agent page and in the downloadable run files.
RESULTS.md — the generated results document for Evaluation v1.0
Each file is copied byte-identical from the project’s own results at build time; the build verifies the copies. Scope: the suite runs against deterministic local fixture pages styled as real product surfaces — the claims are claims about that suite.
Omevra