# Evaluation results

Generated by `scripts/run_eval.py` from `results/run_on.json` and `results/run_off.json`. Every number below comes from those files, which are the raw output of the runs — nothing is hand-entered.

## Summary

- Recovery ON:  **36/36 (100.0%)** tasks passed
- Recovery OFF: **27/36 (75.0%)** tasks passed

## By category

| Category | Tasks | Recovery ON | Recovery OFF |
|---|---|---|---|
| Forms & wizards | 10 | 10/10 (100.0%) | 10/10 (100.0%) |
| Tables, lists & cart | 6 | 6/6 (100.0%) | 6/6 (100.0%) |
| Adversarial recovery | 9 | 9/9 (100.0%) | 0/9 (0.0%) |
| Widgets & dialogs | 5 | 5/5 (100.0%) | 5/5 (100.0%) |
| Business workflows | 3 | 3/3 (100.0%) | 3/3 (100.0%) |
| Portal navigation | 3 | 3/3 (100.0%) | 3/3 (100.0%) |

## Per-task results

| Task | Adversarial | Recovery ON | Attempts (ON) | Recoveries used (ON) | Recovery OFF | Attempts (OFF) |
|---|---|---|---|---|---|---|
| login | no | PASS | 3 | 0 | PASS | 3 |
| table_filter | no | PASS | 2 | 0 | PASS | 2 |
| multistep_form | no | PASS | 7 | 0 | PASS | 7 |
| settings_form | no | PASS | 4 | 0 | PASS | 4 |
| todo_list | no | PASS | 5 | 0 | PASS | 5 |
| sort_table | no | PASS | 1 | 0 | PASS | 1 |
| adversarial_label_change | yes | PASS | 4 | 1 | FAIL | 3 |
| adversarial_overlay_move | yes | PASS | 4 | 1 | FAIL | 3 |
| checkout_full | no | PASS | 11 | 0 | PASS | 11 |
| validation_form | no | PASS | 10 | 0 | PASS | 10 |
| paginated_table | no | PASS | 5 | 0 | PASS | 5 |
| dependent_settings | no | PASS | 6 | 0 | PASS | 6 |
| stacked_modals | no | PASS | 4 | 0 | PASS | 4 |
| delayed_render | no | PASS | 3 | 0 | PASS | 3 |
| date_picker | no | PASS | 2 | 0 | PASS | 2 |
| autocomplete_select | no | PASS | 2 | 0 | PASS | 2 |
| wizard_backtrack | no | PASS | 7 | 0 | PASS | 7 |
| dependent_dropdowns | no | PASS | 3 | 0 | PASS | 3 |
| cart_update | no | PASS | 3 | 0 | PASS | 3 |
| tabbed_panels | no | PASS | 6 | 0 | PASS | 6 |
| accordion_form | no | PASS | 7 | 0 | PASS | 7 |
| inline_edit_table | no | PASS | 3 | 0 | PASS | 3 |
| otp_code_entry | no | PASS | 5 | 0 | PASS | 5 |
| adversarial_double_label_change | yes | PASS | 6 | 2 | FAIL | 2 |
| adversarial_moved_button | yes | PASS | 3 | 1 | FAIL | 2 |
| adversarial_banner_reappears | yes | PASS | 4 | 1 | FAIL | 3 |
| adversarial_late_overlay | yes | PASS | 3 | 1 | FAIL | 2 |
| adversarial_renamed_field | yes | PASS | 4 | 1 | FAIL | 2 |
| adversarial_decoy_button | yes | PASS | 4 | 1 | FAIL | 3 |
| adversarial_rename_under_overlay | yes | PASS | 5 | 2 | FAIL | 3 |
| crm_record_update | no | PASS | 6 | 0 | PASS | 6 |
| vendor_application | no | PASS | 14 | 0 | PASS | 14 |
| lead_entry | no | PASS | 16 | 0 | PASS | 16 |
| portal_customer_drilldown | no | PASS | 3 | 0 | PASS | 3 |
| portal_edit_and_verify | no | PASS | 6 | 0 | PASS | 6 |
| portal_settings_flow | no | PASS | 4 | 0 | PASS | 4 |

## Failures

- [OFF] `adversarial_label_change`: step 3: target_not_found
- [OFF] `adversarial_overlay_move`: step 3: action_error: Locator.click: Timeout 3000ms exceeded.
- [OFF] `adversarial_double_label_change`: step 2: target_not_found
- [OFF] `adversarial_moved_button`: step 2: verify_failed
- [OFF] `adversarial_banner_reappears`: step 3: action_error: Locator.click: Timeout 3000ms exceeded.
- [OFF] `adversarial_late_overlay`: step 2: action_error: Locator.click: Timeout 3000ms exceeded.
- [OFF] `adversarial_renamed_field`: step 2: target_not_found
- [OFF] `adversarial_decoy_button`: step 3: verify_failed
- [OFF] `adversarial_rename_under_overlay`: step 3: target_not_found