OmevraOmevra · Article
The recovery ablation
What the recovery half of the web-agent loop is worth — measured on 36 tasks, with the failures listed and the limits stated.
What was measured
The same agent, the same 36 tasks, run twice in headless Chromium: once with the recovery half of its loop switched on, once with it switched off. 27 tasks are controls — forms, tables, wizards, business workflows and portal navigation. 9 are adversarial: the page mutates or obstructs mid-task — a button is renamed after a field is typed into, an overlay rises over the control, a same-label decoy appears. Every task is a sequence of steps, and every step carries an explicit postcondition written against the page; a task passes only when every postcondition holds. Attempts and recoveries are counted per task in the raw run files (results/run_on.json, results/run_off.json), and every number here is computed from those files.
The result: 36/36 with recovery on, 27/36 with it off
Recovery on passed all 36 tasks. Recovery off passed the 27 controls and failed all nine adversarial tasks. Total attempts: 185 with recovery on, 171 off — on the control tasks the two modes spend identical attempts (148 each), so every extra attempt sits in the adversarial group, where 11 recoveries were spent.
Pass rate by category — recovery on vs off
The whole difference sits in one group. Controls pass either way; adversarial tasks go 9/9 with recovery on and 0/9 with it off.
The adversarial nine — attempts, both modes
Off, each trap task fails in 2–3 attempts. On, each passes in 3–6 — 11 recoveries spent across the nine, none anywhere else.
What recovery does
For each step the agent observes a fresh DOM snapshot, grounds the step's semantic target (role plus text, label or name), acts via the element's locator recipe, and verifies the step's postcondition. When a step fails — target not found, an action error, or a postcondition not met — recovery does three things, each logged:
Re-grounding
The target is looked up again in a fresh snapshot with fallback strategies instead of the exact text or id — for example the submit control of the named form, or the only visible element of that role in scope. That is what survives a button being renamed mid-task (“Pay now” → “Confirm payment”, new id) or moved in the page.
Overlay dismissal
If the click error says another element intercepts pointer events, the agent finds the overlay's dismiss control (Accept / Close / ×), clicks it, and retries.
Retry budget
Retries are budgeted — at most 2 per step — then the task fails and the failure is reported as-is. Recovery off disables all three mechanisms: no re-grounding, no overlay dismissal, no retries — the first failure ends the task. Seven adversarial tasks needed exactly one recovery with recovery on; the two double-mutation tasks needed two.
The nine failures, recovery off
With recovery on there are no failures to list. With it off, the run file records these errors, verbatim:
- adversarial_label_change — step 3: target_not_found
- adversarial_overlay_move — step 3: action_error: Locator.click: Timeout 3000ms exceeded.
- adversarial_double_label_change — step 2: target_not_found
- adversarial_moved_button — step 2: verify_failed
- adversarial_banner_reappears — step 3: action_error: Locator.click: Timeout 3000ms exceeded.
- adversarial_late_overlay — step 2: action_error: Locator.click: Timeout 3000ms exceeded.
- adversarial_renamed_field — step 2: target_not_found
- adversarial_decoy_button — step 3: verify_failed
- adversarial_rename_under_overlay — step 3: target_not_found
Renamed targets were never found, covered buttons' clicks timed out, and decoys failed their postconditions.
Limits, stated plainly
The tasks run on the project's own test pages: deterministic local fixture pages in headless Chromium, styled as real product surfaces (a CRM portal, a store checkout, settings and admin flows) rather than bare HTML. The suite is 36 tasks on those pages — the results are claims about that fixture scope, measured exactly, and not claims about arbitrary live websites.
Full results, charts and the per-task record: web-agent. Method: Evaluation.