OmevraOmevra · Product two
web-agent
An autonomous agent that operates web interfaces end to end — observe, act, verify, recover — measured with an ablation, not adjectives.
Results
The same agent, the same 36 tasks, run twice: once with the recovery half of its loop switched on, once with it switched off.
Twenty-seven of the tasks are controls — forms, tables, wizards, business workflows and portal navigation that a competent agent should pass either way. Nine are adversarial: the page mutates or obstructs mid-task — a button is renamed after a field is typed into, an overlay rises over the control, a same-label decoy appears. Every number on this page is computed from the raw run files of that evaluation (results/run_on.json and results/run_off.json); nothing is hand-entered.
| Category | Tasks | Recovery on | Recovery off |
|---|---|---|---|
| Forms & wizards | 10 | 10/10 (100.0%) | 10/10 (100.0%) |
| Tables, lists & cart | 6 | 6/6 (100.0%) | 6/6 (100.0%) |
| Adversarial recovery | 9 | 9/9 (100.0%) | 0/9 (0.0%) |
| Widgets & dialogs | 5 | 5/5 (100.0%) | 5/5 (100.0%) |
| Business workflows | 3 | 3/3 (100.0%) | 3/3 (100.0%) |
| Portal navigation | 3 | 3/3 (100.0%) | 3/3 (100.0%) |
Recovery on passed all 36 tasks. Recovery off passed the 27 controls and failed all nine adversarial tasks — renamed targets were never found, covered buttons’ clicks timed out, and decoys failed their postconditions. At these sample sizes the intervals are wide (exact Clopper–Pearson 95% CIs): 36/36 is 90.3–100%, 27/36 is 57.8–87.9%, and the adversarial split is 9/9 at 66.4–100% against 0/9 at 0–33.6%.
The results, drawn
Pass rate by category — recovery on vs off
The whole difference sits in one group. Controls pass either way; adversarial tasks go 9/9 with recovery on and 0/9 with it off.
What the outcomes are made of
Recovery on
Recovery off
Recovery on, every task passes — 27 without a single recovery, 9 after recovering. Recovery off, the 9 failures are exactly the adversarial nine.
The adversarial nine — attempts, both modes
Off, each trap task fails in 2–3 attempts. On, each passes in 3–6 — 11 recoveries spent across the nine, none anywhere else.
Attempts per task — the distribution
Total attempts: 185 with recovery on, 171 off. On the 27 control tasks the two modes spend identical attempts (148 each); every extra attempt sits in the adversarial group.
Steps against attempts
9 of the 9 adversarial tasks needed more attempts than they have steps — the points above the line are retries spent recovering. Control tasks sit on the line in both modes.
Every task, both modes
All 36 tasks as one picture: the recovery-on column is green end to end; the recovery-off column turns red on exactly the nine adversarial rows. ↺ counts the recoveries a task used.
Method
The agent drives headless Chromium (Playwright) through the task suite. Each task is a sequence of steps, and every step carries an explicit postcondition written against the page. For each step the agent runs one loop:
Observe and ground
Observe takes a fresh DOM snapshot: every interactive element with a reference, role, text, id, name, label, form and visibility. Ground matches the step’s semantic target — role plus text, label or name — against that snapshot. Act clicks, fills, selects or checks via the element’s locator recipe. Verify checks the step’s postcondition: visible text, row counts, field values, checked state.
Recover — re-grounding
When a step fails — target not found, an action error, or a postcondition not met — recovery first re-grounds: the target is looked up again in a fresh snapshot with fallback strategies instead of the exact text or id — for example the submit control of the named form, or the only visible element of that role in scope. That is what survives a button being renamed mid-task (“Pay now” → “Confirm payment”, new id) or moved in the page.
Recover — overlay dismissal and the retry budget
If the click error says another element intercepts pointer events, the agent finds the overlay’s dismiss control (Accept / Close / ×), clicks it, and retries. Retries are budgeted — at most 2 per step — then the task fails and the failure is reported as-is. Every observe, failure and recovery is written to the run log.
The ablation
Recovery off disables all three mechanisms: no re-grounding, no overlay dismissal, no retries — the first failure ends the task. The evaluation runs the identical suite in both modes, so the difference between the two columns on this page is the measured worth of the recovery half of the loop.
Scope
The evaluation runs against deterministic local fixture pages in headless Chromium — pages styled as real product surfaces (a CRM portal, a store checkout, settings and admin flows) rather than bare HTML. The live session operates the same pages. Seven adversarial tasks needed exactly one recovery with recovery on; the two double-mutation tasks needed two.
Live demo
The live application, embedded — the same fixture pages the evaluation runs on.
web-agent — live application
How to use
1 · Pick a task
Choose one of the 36 fixture tasks — a form, a checkout, a CRM or portal flow, or one of the nine adversarial tasks where the page mutates mid-task. In the live session you can also point the agent at a public page.
2 · Toggle recovery
Run the task with recovery on, then switch recovery off and run the same task again. Recovery off disables re-grounding, overlay dismissal and retries — the first failure ends the task.
3 · Read the results panel and the trace
The results panel reports pass or fail against the task's step postconditions, with attempts and recoveries counted. The trace shows every step as it happens — observe, ground, act, verify, and each recovery — with screenshots along the way.
Evaluation v1.0 — 36-task suite
The current result set on this page is Evaluation v1.0: the full 36-task suite, both modes run. The raw outputs ship with this site as downloadable files, byte-identical to the project's own copies:
- results/run_on.json — the recovery-on run file (36/36 passed)
- results/run_off.json — the recovery-off run file (27/36 passed)
- RESULTS.md — the generated results document
Changelog
v1.0 — the 36-task suite, in six categories:
- Forms & wizards — 10 tasks
- Tables, lists & cart — 6 tasks
- Adversarial recovery — 9 tasks
- Widgets & dialogs — 5 tasks
- Business workflows — 3 tasks
- Portal navigation — 3 tasks
Both modes run for every task: recovery on and recovery off, scored by the same step postconditions.
Full record
Every task, both modes — attempts and recoveries counted from the run files.
| Task | Category | Adversarial | Recovery on | Attempts (on) | Recoveries (on) | Recovery off | Attempts (off) |
|---|---|---|---|---|---|---|---|
| login | Forms & wizards | no | PASS | 3 | 0 | PASS | 3 |
| multistep_form | Forms & wizards | no | PASS | 7 | 0 | PASS | 7 |
| settings_form | Forms & wizards | no | PASS | 4 | 0 | PASS | 4 |
| checkout_full | Forms & wizards | no | PASS | 11 | 0 | PASS | 11 |
| validation_form | Forms & wizards | no | PASS | 10 | 0 | PASS | 10 |
| dependent_settings | Forms & wizards | no | PASS | 6 | 0 | PASS | 6 |
| wizard_backtrack | Forms & wizards | no | PASS | 7 | 0 | PASS | 7 |
| dependent_dropdowns | Forms & wizards | no | PASS | 3 | 0 | PASS | 3 |
| accordion_form | Forms & wizards | no | PASS | 7 | 0 | PASS | 7 |
| otp_code_entry | Forms & wizards | no | PASS | 5 | 0 | PASS | 5 |
| table_filter | Tables, lists & cart | no | PASS | 2 | 0 | PASS | 2 |
| todo_list | Tables, lists & cart | no | PASS | 5 | 0 | PASS | 5 |
| sort_table | Tables, lists & cart | no | PASS | 1 | 0 | PASS | 1 |
| paginated_table | Tables, lists & cart | no | PASS | 5 | 0 | PASS | 5 |
| cart_update | Tables, lists & cart | no | PASS | 3 | 0 | PASS | 3 |
| inline_edit_table | Tables, lists & cart | no | PASS | 3 | 0 | PASS | 3 |
| adversarial_label_change | Adversarial recovery | yes | PASS | 4 | 1 | FAIL | 3 |
| adversarial_overlay_move | Adversarial recovery | yes | PASS | 4 | 1 | FAIL | 3 |
| adversarial_double_label_change | Adversarial recovery | yes | PASS | 6 | 2 | FAIL | 2 |
| adversarial_moved_button | Adversarial recovery | yes | PASS | 3 | 1 | FAIL | 2 |
| adversarial_banner_reappears | Adversarial recovery | yes | PASS | 4 | 1 | FAIL | 3 |
| adversarial_late_overlay | Adversarial recovery | yes | PASS | 3 | 1 | FAIL | 2 |
| adversarial_renamed_field | Adversarial recovery | yes | PASS | 4 | 1 | FAIL | 2 |
| adversarial_decoy_button | Adversarial recovery | yes | PASS | 4 | 1 | FAIL | 3 |
| adversarial_rename_under_overlay | Adversarial recovery | yes | PASS | 5 | 2 | FAIL | 3 |
| stacked_modals | Widgets & dialogs | no | PASS | 4 | 0 | PASS | 4 |
| delayed_render | Widgets & dialogs | no | PASS | 3 | 0 | PASS | 3 |
| date_picker | Widgets & dialogs | no | PASS | 2 | 0 | PASS | 2 |
| autocomplete_select | Widgets & dialogs | no | PASS | 2 | 0 | PASS | 2 |
| tabbed_panels | Widgets & dialogs | no | PASS | 6 | 0 | PASS | 6 |
| crm_record_update | Business workflows | no | PASS | 6 | 0 | PASS | 6 |
| vendor_application | Business workflows | no | PASS | 14 | 0 | PASS | 14 |
| lead_entry | Business workflows | no | PASS | 16 | 0 | PASS | 16 |
| portal_customer_drilldown | Portal navigation | no | PASS | 3 | 0 | PASS | 3 |
| portal_edit_and_verify | Portal navigation | no | PASS | 6 | 0 | PASS | 6 |
| portal_settings_flow | Portal navigation | no | PASS | 4 | 0 | PASS | 4 |
The nine failures, recovery off
With recovery on there are no failures to list. With it off, the run log records:
- adversarial_label_change — step 3: target_not_found
- adversarial_overlay_move — step 3: action_error: Locator.click: Timeout 3000ms exceeded.
- adversarial_double_label_change — step 2: target_not_found
- adversarial_moved_button — step 2: verify_failed
- adversarial_banner_reappears — step 3: action_error: Locator.click: Timeout 3000ms exceeded.
- adversarial_late_overlay — step 2: action_error: Locator.click: Timeout 3000ms exceeded.
- adversarial_renamed_field — step 2: target_not_found
- adversarial_decoy_button — step 3: verify_failed
- adversarial_rename_under_overlay — step 3: target_not_found
Methods and write-up
Methods library
Evaluation — the ablation design and the scoring rules. Calibration — isotonic calibration and per-gene reporting, worst first. Temporal evaluation — the freeze/test design behind the variant results.
The recovery ablation — article
The recovery ablation — what was measured, what recovery does mechanically, the nine recorded failures, and the limits.
Architecture
How the agent is built, from the code itself: the observe → ground → act → verify → recover state machine, the grounding rules, the retry budgets and timeouts, the failure taxonomy, and the safety rails that bound the public-page mode.
Evaluations
The standalone proof page for this product: Evaluations — Evaluation v1.0 in full, with the raw run files as downloads.