OmevraOmevra · Product five
Multi-Agent Evidence
A three-agent system for genetic-variant evidence work — Retriever, Synthesiser, Critic — measured head-to-head against the single-agent baseline it has to beat. Both systems scored at the benchmark’s ceiling — that is the finding.
The system, drawn
Build 1 (the Variant Evidence Agent) showed a single evidence agent can be held to a hard rule — cite, or decline — enforced in code. This project asks the follow-up question: does splitting the work across cooperating agents produce better evidence summaries than one model writing in one pass? Three agents, one guarantee layer, orchestrated as a LangGraph state graph:
- Retriever (deterministic, no model) — gathers the records for a variant from Build 1's local evidence universe: AlphaMissense, the ClinVar record frozen at the evaluation cutoff, the gene prior, and the calibrated model probability. For variants outside that universe it goes live, to ClinVar and PubMed, and marks what it brings back as live context. It assembles the evidence packet.
- Synthesiser (model) — drafts the evidence summary from the packet, one cited claim per bullet, or declines when the packet cannot meet the gate: a calibrated probability plus at least three local evidence records.
- Critic (model) — checks every drafted claim against the packet: does the cited record exist, and does its text support the claim as written, without added strength? Rejected drafts go back with the findings attached; the Synthesiser revises, at most twice.
- Finalise (deterministic, no model) — enforces in code what a prompt can only request: if the gate is not met, the output is the decline whatever the draft said, and any claim citing a record that is not in the packet is removed.
The baseline it has to beat is the same model, the same packet, the same task text, in a single pass — no critic, no revision, no code guard. The packet for each variant was built once and given, identical, to both systems.
The measured comparison — both systems at the ceiling
The evaluation set is a fixed 36 variants: 12 that Build 1's gate summarises, 12 it declines, and 12 real ClinVar variants outside its universe entirely. Both systems are scored by a deterministic checker (does every stated value appear in the record it cites?) plus a manual rubric audit of all 96 claims against their cited records, on citation accuracy, evidence completeness, decline correctness, and cost and latency per variant.
| Measure | Single-agent baseline | Multi-agent system |
|---|---|---|
| Citation accuracy (final, after manual audit of all 96 claims) | 48/48 = 1.00 | 48/48 = 1.00 |
| Citation accuracy (deterministic checker only) | 48/48 = 1.00 | 46/48 = 0.96 |
| Evidence completeness (mean, summaries) | 1.00 | 1.00 |
| Decline correctness (vs the Build 1 gate) | 36/36 | 36/36 |
| Model calls | 36 | 94 |
| Tokens (prompt + completion) | 17,062 | 46,225 |
| Mean latency per variant | 1.40 s | 3.74 s |
| Estimated cost for the 36 variants | $0.0096 | $0.0264 |
Tokens for the 36 variants
Mean latency per variant
The finding is a ceiling effect. On this evaluation the multi-agent system did not beat the baseline on any quality axis — citation accuracy, completeness and decline correctness are identical, both systems at 1.00, the ceiling of what this task, as specified, can measure. A benchmark both systems top out cannot separate them; a harder benchmark is needed. The multi-agent system also cost 2.7× the tokens and 2.7× the latency to get there. That is the result, and it stands.
The 0.96 note: the deterministic checker alone scores the multi-agent system 46/48 = 0.96. The two claims it fails restated a gene prior as a percentage (0.485 → “48.5%”, 0.542 → “54.2%”); the strict checker fails any value not literally in the record. The manual audit rules a fraction and its percentage the same quantity and reclassifies both as supported, which is why the audited figure is 1.00. The baseline never used the percentage form. Both numbers are reported, here and in the project's results.
The task is deliberately well-specified — a small structured packet, an explicit gate rule in the shared task text — and a single strong pass is already reliable in that regime. This evaluation does not show coordination is useless; it shows where its value does and does not appear at this scale. Where it appeared is the next section.
What the critic loop demonstrably did
The critic worked — it just had nothing left to fix in the final outputs. Across the 36 variants it ran 47 rounds and rejected drafts 16 times. Its rejections never changed a final verdict in this run, because finalise enforces the gate regardless — but the loop itself was exercised, counted, and caught real problems in process:
- 13 of 16 rejections policed out-of-universe drafts. There, the packet holds a live ClinVar record with a definite current significance, and the Synthesiser repeatedly drafted as if that live record could carry a summary — or asserted “remains VUS” as a record-backed claim, which the packet does not support either. The critic caught the gate violation every time; on five variants the argument ran the full revision budget (3 critic rounds), after which finalise issued the correct decline anyway. The lesson: persuasion between agents was the expensive path; the code guarantee was the reliable one.
- 3 rejections were false positives on one summarised variant (ASNS), where the critic kept rejecting the phrase “likely pathogenic” for the AlphaMissense class — stricter than the record, which literally carries
class=likely_pathogenic. Two extra rounds spent, outcome unchanged. - The sabotaged-claim catch (failure exercise FM2 below) is the one case where the loop, not the code guard, did the work: a false claim carrying a valid-looking citation, which the guard alone cannot detect.
Failure modes — exercised, not described
| Exercise | Outcome |
|---|---|
| FM1 — Conflicting records: one packet given its frozen ClinVar record (“Uncertain significance”, at the cutoff) and a live ClinVar record (“Likely benign”, evaluated later) | No confusion, but no surfacing either. Zero cross-citation violations — no claim mixed the two statuses up. But the final summary never cited the live record at all: the conflict passed silently. Recorded as a design gap: the architecture has no step whose job is to raise a disagreement between frozen and live records; the critic only reviews claims that exist. |
| FM2 — Critic rejection loop: the Synthesiser sabotaged (exercise-only instruction) into adding an ACMG PS3 claim citing the AlphaMissense record | Handled by the loop. The critic rejected round 1 (unsupported), the revision removed the claim, round 2 accepted. PS3 is absent from the final output. This is the failure class the code guard cannot catch — the citation was real, the claim was not. |
| FM3 — Stale + duplicate evidence: the frozen ClinVar record duplicated (same id twice) plus a superseded older record (“Likely benign”) in one packet | Handled. The final summary claimed the variant's status exactly once, citing the current frozen record; the stale record was never cited. Duplication cannot flip the gate: the gate also requires a model record, which duplicates do not create. |
| FM4 — Forced tool failure: live ClinVar and PubMed forced to raise for an out-of-universe variant | Handled. The run completed and declined, and the outage is visible in the run's event log and the packet's error list. No exception escaped, and no record was fabricated to fill the gap. |
The four exercises cost 10 model calls and 6,583 tokens; their full records ship in the project's eval/results/failure_modes.jsonl.
Two bugs this build found in itself — disclosed
- The ClinVar live parser read the wrong field. The first live implementation read
clinical_significancefrom ClinVar's esummary response; the current API carries the classification ingermline_classification, so live records arrived with significance “not stated”. Found when a tour of the numbers looked empty — before any results were published. Fixed in the live retrieval code; the 12 out-of-universe variants were then re-run with corrected packets (the 24 in-universe pairs never touch live retrieval and stand as first run). This disclosure is also why the critic-rejection counts concentrate on out-of-universe variants: with real live significances in the packets, there was finally something to argue about. - Checker false positive on transcript accessions. The citation checker's numeric extraction initially treated the digits of an accession like NM_005327.7 as a stated value. Fixed by stripping accessions and HGVS tokens before extraction, and verified against the smoke set.
What the whole build cost
All token counts are the API's own usage figures, summed per run — not estimates. Prices are the model's list prices and are labelled estimates wherever used. The evaluation itself (36 variants × 2 systems) used 63,287 tokens ≈ $0.036; the failure-mode exercises 6,583 tokens ≈ $0.004; development smoke tests and probes ≈ 7,400 tokens. The whole build: ≈ 77,000 tokens, ≈ $0.045 of API credit — the multi-agent system at $0.0264 against the baseline's $0.0096 for the same 36 variants.
Downloads
The project's own results document, byte-identical to the file in the project:
- RESULTS.md — the full results document behind every figure on this page: the comparison, the critic log, the failure-mode record, the token accounting, the disclosed bugs and the limitations
Limitations
- One run per variant per system. No variance estimate; a different sampling of the model could move individual claims. The comparison is of these runs, not of distributions.
- API-default temperature. The model-access helper's interface does not expose temperature, so all calls ran at API defaults. Prompts, packets and the eval set are fixed and shipped, which bounds — but does not eliminate — this.
- Ceiling regime. Packets hold 1–4 short records and the gate rule is stated in the task text. The interesting differences between the architectures appeared in process (rejections, sabotage, conflicts), not in final-output quality. A harder regime — larger, messier packets, conflicting sources as the norm — is where a difference would be expected to show, and is the obvious next evaluation.
- Live data moves. Out-of-universe packets were retrieved from ClinVar and PubMed at run time; re-running later may fetch updated records. The frozen-universe results are exactly reproducible; the live ones are reproducible in method, not to the byte.
- The manual audit is one reader, applying the rubric shipped with the project; its two reclassifications are individually documented in the audit file.
Every figure on this page is taken from the project's RESULTS.md and its evaluation result files, and is cross-checked against both at build time.
Evaluations
The standalone proof page for this product: Evaluations — the protocol, the versioned results, and every result file as a download.