OmevraOmevra · Methods
Evaluation
The ablation design and scoring rules behind the web-agent evaluation — in the project's own terms.
Ablation design
The evaluation runs the identical 36-task suite twice in headless Chromium (Playwright): once with the recovery half of the agent's loop switched on, once with it switched off. Recovery off disables all three recovery mechanisms — no re-grounding, no overlay dismissal, no retries — so the first failure ends the task. The difference between the two runs is the measured worth of recovery.
For each step of a task the agent runs one loop: observe (a fresh DOM snapshot — every interactive element with a reference, role, text, id, name, label, form and visibility), ground (match the step's semantic target — role plus text, label or name — against that snapshot), act (click, fill, select or check via the element's locator recipe), verify (check the step's postcondition), and, on any failure, recover.
Scoring rules
Postconditions decide pass and fail
Every step of every task carries an explicit postcondition written against the page: visible text, row counts, field values, checked state. A task passes only when every step's postcondition holds. There is no partial credit and no judgement call after the run.
Attempts and recoveries are counted, not estimated
Each run writes a raw run file (results/run_on.json and results/run_off.json) recording per task: success, steps completed and total, attempts, recoveries used, and the recorded error for a failure. Retries are budgeted — at most 2 per step — then the task fails and the failure is reported as-is. Every observe, failure and recovery is also written to the run log.
Controls and adversarial tasks
27 of the 36 tasks are controls — forms, tables, wizards, business workflows and portal navigation that a competent agent should pass either way. 9 are adversarial: the page mutates or obstructs mid-task — a button is renamed after a field is typed into, an overlay rises over the control, a same-label decoy appears. The suite by category: Forms & wizards (10) · Tables, lists & cart (6) · Adversarial recovery (9) · Widgets & dialogs (5) · Business workflows (3) · Portal navigation (3).
| Category | Tasks | Recovery on | Recovery off |
|---|---|---|---|
| Forms & wizards | 10 | 10/10 (100.0%) | 10/10 (100.0%) |
| Tables, lists & cart | 6 | 6/6 (100.0%) | 6/6 (100.0%) |
| Adversarial recovery | 9 | 9/9 (100.0%) | 0/9 (0.0%) |
| Widgets & dialogs | 5 | 5/5 (100.0%) | 5/5 (100.0%) |
| Business workflows | 3 | 3/3 (100.0%) | 3/3 (100.0%) |
| Portal navigation | 3 | 3/3 (100.0%) | 3/3 (100.0%) |
Result of the current evaluation set: recovery on 36/36, recovery off 27/36; adversarial tasks 9/9 with recovery on and 0/9 with it off; controls 27/27 both ways. Full charts and the per-task record are on the web-agent page; the ablation is written up in the recovery ablation.
Scope
The evaluation runs against deterministic local fixture pages in headless Chromium — pages styled as real product surfaces (a CRM portal, a store checkout, settings and admin flows) rather than bare HTML. The claims on this site are claims about that suite.
Related method pages: Calibration · Temporal evaluation. Products: web-agent · Variant Evidence Agent.
Architecture
How the systems are built
web-agent architecture — the state machine, grounding rules, budgets and the public-page rails, from the code. Variant Evidence Agent architecture — retrieval, scoring, calibration, and the cite-or-decline gate enforced in code.