OmevraOmevra · Multi-Agent Evidence — proof
Multi-Agent Evidence — evaluations
Evaluation v1 in full: the ceiling result, the cost of reaching it, and the complete evaluation record.
Evaluation v1 — 36 variants, two systems, one packet each
Version label: Evaluation v1 — fixed set of 36 variants. The set (seeded selection, eval_set.csv) holds 12 variants the Variant Evidence Agent's gate summarises, 12 it declines, and 12 real ClinVar variants outside its universe entirely. The evidence packet for each variant was built once and given, identical, to both systems: the multi-agent system (Retriever → Synthesiser → Critic, with a deterministic finalise step enforcing the gate in code) and the single-agent baseline — the same model, the same packet, the same task text, one pass. Scoring is a deterministic checker (every stated value must appear in the record it cites) plus a manual rubric audit of all 96 claims against their cited records. Completeness is the fraction of packet records cited, scored on summaries; decline correctness is agreement with the gate on all 36.
Results — both systems at the ceiling
| Measure | Single-agent baseline | Multi-agent system |
|---|---|---|
| Citation accuracy (final, after manual audit of all 96 claims) | 48/48 = 1.00 | 48/48 = 1.00 |
| Citation accuracy (deterministic checker only) | 48/48 = 1.00 | 46/48 = 0.96 |
| Evidence completeness (mean, summaries) | 1.00 | 1.00 |
| Decline correctness (vs the Build 1 gate) | 36/36 | 36/36 |
| Model calls | 36 | 94 |
| Tokens (prompt + completion) | 17,062 | 46,225 |
| Mean latency per variant | 1.40 s | 3.74 s |
| Estimated cost for the 36 variants | $0.0096 | $0.0264 |
The headline is a ceiling effect. After the manual audit, citation accuracy is 48/48 for the baseline and 48/48 for the multi-agent system — 1.00 against 1.00, the top of what this benchmark can measure, so the test cannot separate the two systems and a harder benchmark is needed. The deterministic checker alone scored the multi-agent system 46/48 = 0.96 (two claims restated a gene prior as a percentage; the audit rules a fraction and its percentage the same quantity). Completeness is 1.00 both ways and decline correctness 36/36 both ways. The multi-agent system reached that ceiling at higher cost: 46,225 tokens against 17,062 per variant set (94 LLM calls against 36), and mean latency 3.74 s against 1.40 s per variant — 2.7. Nothing here is dressed as a win.
What the coordination demonstrably did is in the process record: the Critic ran 47 rounds and rejected drafts 16 times — 13 of the rejections on out-of-universe variants, and 3 rejections that were one false positive on a single summarised variant. The failure-mode exercises (conflicting records, a sabotaged draft, stale + duplicate evidence, forced tool failure) are recorded in the downloads.
Downloads — the evaluation record
- RESULTS.md — the full results document
- final_metrics.json — final metrics (checker merged with the manual audit)
- metrics.json — deterministic-checker metrics
- manual_review.csv — the manual audit of all claims
- failure_modes.jsonl — the failure-mode exercise records
- eval_set.csv — the fixed 36-variant evaluation set
Each file is copied byte-identical from the project's eval/ outputs at build time; the build verifies the copies.
Product: Multi-Agent Evidence.