Variant Evidence Agent

Variants of Uncertain Significance (VUS) are the daily bottleneck of clinical genomics.

Predictors disagree. Most published models are trained and tested on random ClinVar splits — which leaks variants that were reclassified using the same evidence the model is then credited for learning. And an AUROC of 0.99 tells a lab nothing about whether a “90% pathogenic” call is right 90% of the time.

The gap is not another AUROC point. The gap, as the project’s README states it, is three things at once:

• Temporal validity — train on classifications frozen at a cutoff, test only on variants reclassified after it.
• Calibration — a probability that means what it says, reported per gene family (ECE / Brier), including where calibration fails.
• Evidence provenance — every claim in a report cited to a retrieved record, with an explicit allowed answer: “insufficient evidence — remains VUS.”

All data is open and pinned by date, because the temporal split depends on it. The cutoff snapshot held 2,366,650 GRCh38 rows; the current snapshot holds 4,578,190. The split report (data/processed_v2/SPLIT_REPORT.md) records exactly what fell on each side of the freeze:

• Train — 1,103,657 variants labelled at the Jan 2024 cutoff and frozen: 240,752 pathogenic, 862,905 benign.
• Test — 27,748 variants that were uncertain at the cutoff and have since been reclassified: 8,856 pathogenic, 18,892 benign. No test label existed at train time.
• Near-duplicate guard: test rows sharing gene + name with train are removed — the guard removed 0 rows.

The split the whole evaluation rests on, as one picture. Classifications are frozen at the Jan 2024 cutoff; the test set is only what changed afterwards — variants whose labels did not exist when the model was trained:

Source: data/processed_v2/SPLIT_REPORT.md and docs/validation-dossier.md §2, §7. Of the 27,748 post-freeze reclassifications, the agent summarises 10,357 (37.3%) with cited evidence and declines 17,391 (62.7%). Coverage is the binding limit: the predictors are missense-only, so most of the declined variants are non-missense variants with no predictor score at all. A decline is a designed output, not an error.

All rows are on the same temporal test set — variants reclassified after the Jan 2024 freeze. Lower ECE and Brier are better; higher AUROC is better. This is the README’s results table, from the model reports in data/processed_v2/:

Read the middle rows carefully: adding the gene prior lifted AUROC to 0.959 but the raw ensemble’s calibration worsened to ECE 0.251. Discrimination and calibration are different claims — the table reports both, for every variant of the model, including the ones that got worse. The final row changed on 2026-10-26: Model Report 6’s head-to-head adopted GBM+R for the best AUROC (0.970) and Brier (0.062) of the field — at a calibration cost: its ECE of 0.025 is worse than the earlier logistic model’s 0.017, which stays the best-calibrated row in this table. That comparison is kept in full on the evaluations page.

Further variants from the model reports

Model Reports 2–4 score additional variants under the same protocol. None replaces the headline table; all are published:

The REVEL ablation (Model Report 4): on the test set REVEL covers 39.7% of variants, AlphaMissense 37.4%, either 39.8% — both predictors are missense-only, so REVEL barely widens coverage (+0.1 pt combined). Its value came from signal, not reach: AUROC 0.958 → 0.968 and Brier 0.078 → 0.066 at essentially unchanged calibration. The MLP calibrates about as well after isotonic (ECE 0.013) but ranks worse (AUROC 0.947) and scores worse overall (Brier 0.090).

Calibration by gene — final model

Isotonic-calibrated ECE per gene for the final model (GBM+R — gradient boosting + REVEL + isotonic), genes with n ≥ 50 scored test variants, from Model Report 6. Fourteen of the fifteen largest genes sit between 0.005 and 0.154; USH2A, at 0.204, does not — and stays in the table. Against M2+R, USH2A improved from 0.290 while MUTYH degraded 0.083 → 0.154 and NF1 moved 0.117 → 0.130:

For scale: under AlphaMissense alone with Platt calibration, per-gene ECE across these same genes ran 0.203–0.428 (Model Report 1). Under the isotonic-calibrated ensemble before REVEL, most genes sat at 0.020–0.128, with NF1 (0.231) and USH2A (0.278) the named failures (Model Report 2).

Model Report 1 states the principle the whole project runs on: the point of calibration is that ECE and Brier improve and the number means what it says — and if they don’t improve, that is reported too. On the temporal test, the first calibration attempt did not improve. Platt scaling fitted on the 2024 training window was applied to variants reclassified after the freeze, and the probabilities got worse:

The failure only appears because the test set is temporal. A random split grades the model partly on evidence it has already seen, and a calibration map that does not transfer forward in time can still look fine. The split design — freeze, then test only on what was reclassified afterwards — is what makes the failure, and then the improvement, visible at all.

The reports also record the project catching its own error: the first committed version of Model Report 3 printed ECE values computed with an argument-order bug in the ECE call (labels passed where probabilities belong). A reproduction audit of Model Report 2 caught it; the affected values were recomputed, AUROC and Brier were unaffected, and the correction note is kept in the report.

The calibration story above, as charts — computed from the stored per-variant predictions (data/processed_v2/test_predictions*.csv and, for the adopted final model, headtohead_20261026_predictions.csv), binned exactly as the project’s ECE is binned. Two reliability series are drawn from the files: the Platt failure and the final model. The raw ensemble’s per-variant predictions were never stored, so its ECE appears as a figure, not a curve — the same note the model analysis section keeps.

Reliability — predicted against observed

AlphaMissense + Platt (fit on 2024) sits far off the diagonal — ECE 0.157, worse than the uncalibrated score’s 0.040. The final model (GBM+R — gradient boosting + REVEL + isotonic, adopted 2026-10-26) stays on it — ECE 0.025, AUROC 0.970, Brier 0.062, recomputed from headtohead_20261026_predictions.csv (prob_C) and matching Model Report 6.

Calibration error across the ablation

Expected calibration error for the six rows of the results table (source: docs/validation-dossier.md §3 and its §7 addendum). Adding the gene prior made the raw ensemble the worst-calibrated model on the page — ECE 0.251. Isotonic calibration fitted on the train holdout brought the ensemble to 0.017; the final model, GBM+R, reports 0.025.

Calibration by gene — worst first

The final model’s per-gene ECE, the 24 genes with n ≥ 50 scored test variants, ordered worst first — the ordering the tables do not use:

Source: docs/validation-dossier.md §7 addendum (Model Report 6) — every value recomputed from headtohead_20261026_predictions.csv (prob_C) and matching. 23 of the 24 genes sit between 0.005 and 0.154; USH2A, at 0.204, is a calibration failure by the project’s own standard and is drawn in red, on the same axis as every other gene.

The tables above are the headline record. This section draws the same record as charts — every curve computed from the per-variant prediction files the model reports published (data/processed_v2/test_predictions*.csv), never re-fitted for display. The chart data ships beside this page as model_analysis_data.json; how each figure is derived, and from which file and column, is documented in the repo (docs/model-analysis.md).

Model Report 5 adds two more model families under the identical protocol — same features (AlphaMissense, gene prior, REVEL), same seed-11 train split, isotonic fitted on the train holdout only: gradient boosting and random forest (scikit-learn). Both land at AUROC 0.970 against the logistic M2+R model’s 0.968, with Brier 0.062–0.063 against 0.066 and ECE 0.024–0.025 against 0.023. Model Report 6 (2026-10-26) then re-ran every candidate head-to-head on the same frozen test and adopted the gradient-boosting model (GBM+R) as the final model — the headline above; RF+R stays in the charts as the close second it was.

Loading chart data…

Reliability diagram

Mean predicted probability against the observed fraction of pathogenic labels, in 10 equal-width bins — the binning the project’s ECE uses. Points sit on the diagonal when a probability means what it says; each point is labelled with the number of test variants in its bin. Isotonic-calibrated models emit few distinct probabilities, so they show fewer, larger bins.

ROC curve

True-positive rate against false-positive rate, swept over the stored scores of the same model variant.

Ablation — the full table as bars

All seven ablation rows: the six from the results table above, plus the random-forest model from Model Report 5 as the seventh. Each row is labelled with its own n — 10,373 for the AlphaMissense-based models, 10,357 for the models that also require REVEL. All three metrics live on a 0–1 scale; lower ECE and Brier are better, higher AUROC is better.

Sources: rows 1–5, Model Reports 1, 2 and 4; row 6, Model Report 5’s gradient boosting, adopted as the final model in Model Report 6; row 7, Model Report 5 (data/processed_v2/MODEL_REPORT_5.md). The raw ensemble row is charted from the published table — its per-variant predictions were never stored, so it has no curve above.

Calibration by gene — final model

Isotonic-calibrated ECE per gene for the final model (GBM+R), the fifteen genes of Model Report 4’s table with their GBM+R values (Model Report 6). USH2A, the worst gene at 0.204, is drawn in red and kept on the same axis as every other gene.

Source: data/processed_v2/MODEL_REPORT_6.md and headtohead_20261026.json — GBM+R isotonic-calibrated ECE by gene, test set, n ≥ 50. Model Report 5 reports the same fifteen genes for the two tree models.

The agent sits on top of the calibrated model. Retrieval runs over frozen ClinVar records and predictor records; the agent drafts an ACMG-style summary in which every claim is cited to a retrieved record — the cite-or-decline rule is enforced in code (src/vus_agent/agent.py), not left to the model’s discretion. Where the retrieved evidence does not support a summary, the allowed output is a decline: “insufficient evidence — remains VUS.” An adversarial test in the repo (tests/test_agent_refuses.py) feeds the agent a fabricated citation and a missing record; it must decline both.

The decline rate follows from coverage: the agent may summarise only variants with a retrieved predictor record, and 62.7% of the test set has none the final model can score. The reports file (examples/agent_reports_real.md, generated by scripts/run_agent_real.py) publishes real outputs from the test set — including the confident calls, the declines, and the misses. Two of them:

NM_000492.4(CFTR):c.1373G>T (p.Gly458Val)
Verdict: evidence summary (not a clinical classification)
Calibrated P(pathogenic): 0.999

Evidence (every claim cited):
  [AlphaMissense] score=0.961 class=likely_pathogenic <AM:7:117548804:G>T>
  [ClinVar (frozen 2024-01)] status then: Uncertain significance <CLINVAR:allele22171>
  [Gene prior (frozen train)] gene=CFTR <GENEPRIOR:CFTR:frozen2024>

(Future truth, evaluation only: reclassified “Likely pathogenic”.)

The prediction, the calibrated probability, and the three retrieved records it rests on — nothing else is claimed. Later labels are evaluation-only and are never agent evidence.

NM_000274.4(OAT):c.192_193del (p.Gly65fs)
Verdict: insufficient evidence — remains VUS

Evidence (every claim cited):
  [ClinVar (frozen 2024-01)] status then: Conflicting interpretations
  of pathogenicity <CLINVAR:allele15217>

A frameshift with no predictor score and one conflicting record gets silence, not a story.

The same file also publishes the agent’s misses. Under the earlier logistic model, three variants (HOGA1, PROC, GLB1) were scored at P = 0.493 whose later labels were pathogenic; under GBM+R two of them are resolved (HOGA1 0.808, GLB1 0.979), PROC remains under-confident at 0.464, and the benign-leaning RANBP2 summary remains a confident miss at 0.006 for a variant later reclassified pathogenic. They are kept in the reports alongside the correct calls.

These are the agent’s actual outputs on the post-freeze test set — all 27,748 of them, produced by scripts/run_agent_real.py from the frozen evidence: 10,357 cited evidence summaries and 17,391 declines. Search by gene symbol (for example CFTR) or by variant text (for example c.1373G>T). If a variant is not in the test set, there is no stored report — the agent has never seen it, and nothing here is reconstructed.

Type at least 2 characters to search the stored reports.

The README’s limits section is headed “say them out loud.” They are:

• Temporal reclassification labels are themselves partly predictor-influenced. The temporal split removes the direct leak; it does not make labels pure. That circularity is discussed, not hidden.
• AlphaMissense scores cover missense only, precomputed only. Weights are not released, so there is no novel-variant scoring beyond the precomputed predictions — which is also why the agent declines the majority of the test set.
• A calibrated probability is not a clinical classification. ACMG classification remains a human, guideline-driven judgement.
• gnomAD allele frequency is not in the model. The downloaded allele-number file is locus-level AN only and cannot yield AF; the project ships without AF rather than relabelling coverage as frequency, with the VCF/API route documented as the deferred path.
• dbNSFP was deferred, and the scores are linked, not redistributed. Bulk dbNSFP download needs institutional-email registration; REVEL was taken from its public Zenodo record instead. AlphaMissense licence wording is ambiguous between sources, so the score file is linked to, never redistributed.

The repo assembles its validation evidence into a single dossier (docs/validation-dossier.md), drawn entirely from the project’s own split, feature, and model reports. It covers the same ground a documentation package for a clinical-grade system carries — intended use, validation design, performance including the worst cases, failure modes, and the oversight design — in the project’s own terms:

The project runs on CPU with fully public data, and a Dockerfile is provided. The demo needs no downloads and runs in about ten seconds — it uses a small synthetic variant table to show the full loop: features → calibrated probability → cited evidence summary → and the refusal path when evidence is insufficient:

PYTHONPATH=src python3 -m vus_agent.demo
PYTHONPATH=src python3 tests/test_agent_refuses.py

The full pipeline reproduction — the scripts behind every number on this page — runs from the downloaded ClinVar and AlphaMissense files:

python3 scripts/build_temporal_split.py
python3 scripts/join_alphamissense.py
python3 scripts/train_ensemble.py        # Model Report 2: AUROC 0.958, ECE 0.017
python3 scripts/join_revel.py            # REVEL v1.3, public Zenodo record
python3 scripts/train_revel_ablation.py  # Model Report 4: AUROC 0.968
python3 scripts/eval_headtohead_20261026.py  # Model Report 6: GBM+R adopted, AUROC 0.970
PYTHONPATH=src python3 scripts/run_agent_real.py

The Docker image (python:3.12-slim) installs the requirements, copies the source, and runs the demo as its command. Real run output — the demo, the agent on real variants, and a refusal — is collected in examples/DEMO.md.

The live application, embedded. The standalone application stays at evidra-variant-evidence-agent.hf.space.

How the system is built, from the code itself: retrieval from prediction-time records, the stored model probability, calibration fitted on train data only, the cite-or-decline gate, live-evidence mode, and the batch flow.

Variant Evidence Agent architecture ›

The full proof record for this product: Evaluations — the temporal-split protocol, versioned results v1 to v6, per-gene calibration worst first, the before/after decline record, and every result file as a download. The public API is documented at API documentation.