OmevraOmevra · web-agent
web-agent architecture
The state machine, the modules, the budgets — and the rails that bound the public-page mode. Drawn from the code itself.
The loop, as a state machine
One task is a list of steps. For each step the agent runs the attempt loop drawn below — this is Agent.run_task and Agent._run_step in src/agent.py, not a paraphrase of them. The agent the evaluation measures and the agent in the live application are the same code: the app imports src.agent.Agent, and its StreamingAgent subclass adds screenshots after each step and nothing else — its own docstring reads “Behaviour identical.”
Every transition is written to a JSONL run log (RunLog in src/agent.py): observe (with the snapshot's element count), step_ok, failure (with its class), recovery, and task_success (with the attempt and recovery totals). The first step that exhausts its budget ends the task; steps already completed stay counted.
The modules — who owns what
src/snapshot.py — observation
Snapshot.take runs an injected script (SNAPSHOT_JS) that enumerates a, button, input, select, textarea, [role="button"], [role="link"] and records each element's tag, type, role, id, name, visible text (trimmed to 80 characters), value, placeholder, aria-label, its label (from label[for] or an enclosing label), its form's id, up to 6 ancestor ids, and whether it is visible (bounding box plus computed style) and enabled. The snapshot then assigns each record a ref (e1, e2, …) and a locator recipe, built by locator_recipe in a fixed order: #id first, then tag[name="…"], then tag:has-text("…") for buttons and links, then [placeholder="…"]. An element matching none of these gets no recipe — and an action on it fails as an action error, in the open.
src/agent.py — the loop
The task model is three dataclasses: Target (role, plus optional text, label, name, enclosing-form hint and a kind hint such as submit), Step (an action — click, fill, select or check — a target, a value, and an optional verify callable), and Task (a fixture page and its steps). Agent owns grounding (_find_exact, _find_fallback), acting (_act), verification (_verify), overlay dismissal (_dismiss_overlay) and the attempt loop itself. TaskResult accumulates success, steps completed, attempts and recoveries across the whole task — those accumulators are the numbers in the run files.
src/tasks.py — the suite
The 36 tasks and their postconditions. Each step's verify callable is built from small helpers in the same file — text_is, text_contains, field_value_is, visible_rows, first_cell_is, cell_text_is, li_count_is, is_checked, element_visible, element_absent, element_count_is, and all_of to combine them. A step may carry no postcondition; then a completed action completes the step.
scripts/run_eval.py — the measurement
Runs the suite twice — recovery on, recovery off — writes results/run_on.json and results/run_off.json plus the JSONL logs, and generates RESULTS.md from the run files. The evaluation method page covers the design and the scoring rules.
The live application — runner.py and app.py
StreamingAgent subclasses Agent and adds one thing: a viewport screenshot (1100×750, full_page=False) after each step and after an overlay dismissal — a failed capture never fails the run. The server runs each run on its own thread (Playwright's sync API is thread-bound) and streams the log events to the page as they happen.
Grounding — exact first, structural on recovery
Exact grounding (_find_exact) considers only visible, enabled elements whose role matches — a button is a <button>, a role="button" element, or a submit/button input; a textbox is a <textarea> or a text-like input; a combobox is a <select>; checkbox and radio match their input types; a link is an <a>. Scoring is blunt: an exact text match scores 100, a substring match 50; when the target names a label or a name attribute, a candidate that does not match it exactly is rejected outright. The highest-scoring candidate wins.
Fallback grounding (_find_fallback) runs only inside recovery, when the target may have been renamed or moved. In order: take the visible, enabled candidates of the role; if the target names a form or container, narrow to the elements inside it (by form id or ancestor ids) — if the target's kind is submit and exactly one submit/button control is in scope, take it; if exactly one element is in scope, take it; as a last resort, if exactly one visible candidate of that role exists on the whole page, take it. Otherwise there is no match, and the failure stands. Structure is allowed to answer only when it answers unambiguously.
Retries, budgets, timeouts
The retry budget
MAX_RETRIES = 2 per step, and only with recovery on — with recovery off the budget is 0, so the first failure ends the task. Each retry taken is counted as one recovery in the task's totals, whether or not it is the retry that succeeds. When the budget is spent, the step returns its failure class and the task ends with the error step N: <class>.
Timeouts
Every action — click, fill, select, check — carries a 3,000 ms timeout (ACTION_TIMEOUT_MS). Verification polls the step's postcondition every 0.1 s for up to 2.0 s (VERIFY_TIMEOUT_S) before declaring verify_failed. The overlay-dismiss click gets 1,500 ms. Nothing in the loop waits indefinitely.
Concurrency and the end of a run
The application bounds runs rather than offering a mid-run cancel. At most 2 runs execute at once — a semaphore in app.py; a further request is refused immediately (HTTP 429, “another run is in progress”). Events stream from the run thread through a queue; a stream that sees no event for 180 seconds is closed with a “run timed out” note. If the visitor closes the stream, the run finishes on its own thread, as the server code puts it. And every run finishes by emitting exactly one terminal event — a verdict or an error: a harness-level failure is reported as an error, never dressed up as a verdict.
The failure taxonomy — what breaks, what answers it
The loop records exactly three failure classes. The counts below are the recovery-off run's nine failures, classified from the error strings recorded in run_off.json — the same strings the ablation article lists verbatim.
| Failure class | Off-run failures | The recovery that answers it |
|---|---|---|
target_not_found | 4 of 9 — adversarial_double_label_change, adversarial_label_change, adversarial_rename_under_overlay, adversarial_renamed_field | Re-grounding: a fresh snapshot, then the structural fallbacks — the target was renamed or moved, so exact text and id are abandoned on purpose. |
action_error (click timeout, 3,000 ms) | 3 of 9 — adversarial_banner_reappears, adversarial_late_overlay, adversarial_overlay_move | Overlay dismissal, when the error reports another element intercepting pointer events: find a visible button whose text is in the dismiss set (×, Close, Accept, Accept all, Dismiss, Got it, OK, No thanks), click it, then retry the step. |
verify_failed | 2 of 9 — adversarial_decoy_button, adversarial_moved_button | Re-ground inside the named form's scope and retry. The postcondition is the arbiter: a decoy's click succeeds as an action and is caught here, by what the page does not show afterwards. |
With recovery off, all three classes are terminal at first occurrence — that is the ablation. With it on, the same nine tasks pass, spending 11 recoveries between them.
Permissions and untrusted input — the public-page mode
In the live application a visitor supplies a URL, a label, and — for one of the two intents — a text to type. All of it is untrusted input. The rails around it are code: src/public_mode.py (validation and field rules, written as pure functions) and PublicStreamingAgent / execute_public_run in the app's runner.py (enforcement during the run). The loop inside the rails is the loop above, unchanged.
Validation before any browsing
validate_public_url accepts only http(s) URLs. Localhost names and .local names are refused by name. The host is then resolved, and every address it resolves to is checked: a single private, loopback, link-local, reserved, multicast or unspecified address refuses the URL, before a browser is opened.
Two fixed intents, a hard step budget
The visitor can ask for exactly two shapes of work: click the element with this label, or type this text into the field with this label and submit. The steps are built from a fresh snapshot by _public_steps and capped at MAX_PUBLIC_STEPS = 2 — deliberately inside the fixture suite's own budgets (fixture tasks run up to 16 steps; the recorded fixture maximum is 16 attempts for a single task). The visitor's inputs select a target; they cannot add steps, change the postconditions, or raise the budget.
The domain lock
Validation computes the entered host's registrable domain (eTLD+1, with common second-level public suffixes handled, so example.co.uk is one domain). During the run, a network route guard aborts any navigation request to a different registrable domain — subdomains of the entered domain are fine — and records the blocked host; the agent additionally checks the page's domain before and after every step. Leaving, or trying to leave, is a hard stop: the verdict names the domain it was stopped for. A blocked navigation is reported, never hidden.
Field refusals
field_block_reason names the fields the agent must never type into: password fields, always; and any field whose label, name, placeholder, aria-label or id contains a payment term — card, cvv, cvc, expiry, expiration, billing, payment, credit, debit, iban. The check runs when the steps are built and again before every fill step, and a refusal ends the run with the reason stated.
Downloads off
The public run's browser context is created with downloads disabled, and any download the page attempts is cancelled on the event.
Scope
The measured suite runs on deterministic local fixture pages in headless Chromium, styled as real product surfaces. The public-page mode is bounded exactly as described above. Results: web-agent · Method: Evaluation · Write-up: The recovery ablation.
This page describes src/agent.py, src/snapshot.py, src/tasks.py, src/public_mode.py and scripts/run_eval.py in the project, and runner.py / app.py in the live application — the code as it stands.