Variant Evidence Agent — evaluations

Version label: Evaluation v1 — temporal split, train frozen Jan 2024. This page is the proof record for the Variant Evidence Agent: the protocol in full, every versioned result, and the files behind them.

All data is open and pinned by date. ClinVar classifications are frozen at the Jan 2024 cutoff: the cutoff snapshot held 2,366,650 GRCh38 rows (the current snapshot holds 4,578,190). The train set is the 1,103,657 variants labelled at the cutoff (240,752 pathogenic, 862,905 benign), frozen. The test set is only the 27,748 variants that were of uncertain significance at the cutoff and have since been reclassified (8,856 pathogenic, 18,892 benign) — no test label existed at train time. Features are joined as of the cutoff where the source allows dating; calibration maps are fitted on train data only (an 80/20 seed-11 split, isotonic fitted on the train holdout) and judged on the untouched test set. The near-duplicate guard removes test rows sharing gene + name with train; it removed 0 rows in this build. Full design: Temporal evaluation · Calibration.

Each model report is a version, kept — including the versions that lost. Metrics are AUROC / ECE / Brier on the temporal test set, recomputed at build time from the stored prediction files and checked against the reports.

Evaluation v3 model adoption, 2026-10-26: the current adopted model is GBM+R (Report 6), chosen by the decision rule fixed before the head-to-head run — highest isotonic AUROC, ties by Brier then ECE. It has the best AUROC (0.970) and best Brier (0.062) of the field. The trade-off: GBM+R's ECE 0.025 is worse than the earlier logistic model’s 0.017 (Model 2, Report 2 — the model the live app served until 2026-10-26), which stays the best-calibrated row in the table, and worse than M2+R's 0.023. M2+R (Report 4) and Model 2 (Report 2) remain in the table as labelled history with their real metrics. The Platt row is a calibration failure kept in the record: fitting on 2024 moved ECE from 0.040 to 0.157 at unchanged AUROC.

Reliability — predicted against observed

Calibration error across the ablation

The final model's (GBM+R, Report 6) per-gene ECE for the 24 genes with n ≥ 50 scored test variants, ordered worst first. 23 genes sit between 0.005 and 0.154; USH2A at 0.204 is a calibration failure by the project's own standard and leads the table. The adoption was not uniform: USH2A improved from M2+R's 0.290, while MUTYH degraded 0.083 → 0.154 and NF1 moved 0.117 → 0.130. The per-gene tables are inside the downloadable model reports (Reports 4 and 5 carry the earlier models' tables in full; Report 6 carries GBM+R's).

The agent's contract is cite-or-decline: a summary is issued only where the evidence supports one, and every other variant is declined individually. On the temporal test set the live split is 10,357 summarised (37.3%) against 17,391 declined (62.7%). Coverage is the binding limit: the model scores 37.3% of the test set — the missense variants with predictor scores — and most of the declined remainder are non-missense variants with no predictor score at all. Extending coverage to splice and non-coding variants is next on the roadmap.

Before / after — the 2026-10-26 model adoption

Before — Model 2 served (logistic, AlphaMissense + gene prior). The output split the live app actually served until 2026-10-26, as published in docs/validation-dossier.md §4–§5 before the addendum:

After — GBM+R live (gradient boosting + REVEL + isotonic; Evaluation v3 model adoption, 2026-10-26). The cite-or-decline rule did not change (MIN_EVIDENCE = 3, unchanged). The counts shifted only because GBM+R requires a REVEL score in addition to AlphaMissense: 16 variants carry an AlphaMissense score but no REVEL score, so they no longer receive a model probability and are declined like every other unscored variant:

Source: data/processed_v2/MODEL_REPORT_6.md — before 10,373 summarised / 17,375 declined; after 10,357 summarised / 17,391 declined, of 27,748. The after counts are the live counts: the app's /api/meta reports summarised 10,357 / declined 17,391 with model version “v3 gradient boosting (GBM+R)”.

Each file below is copied byte-identical from the project's data/processed_v2/ (and the validation dossier from docs/) at build time; the build verifies the copies.

• MODEL_REPORT.md — Report 1: AlphaMissense alone, temporal test
• MODEL_REPORT_2.md — Report 2: ensemble (AlphaMissense + gene prior)
• MODEL_REPORT_3.md — Report 3: from-scratch MLP (does not beat the ensemble)
• MODEL_REPORT_4.md — Report 4: REVEL ablation — the headline M2+R model, with its per-gene table
• MODEL_REPORT_5.md — Report 5: tree ensembles, with the per-gene comparison table
• MODEL_REPORT_6.md — Report 6: the final-model head-to-head and the adoption of GBM+R (2026-10-26)
• SPLIT_REPORT.md — the temporal split report
• FEATURE_REPORT.md — the feature report
• validation-dossier.md — the full validation dossier behind this page
• test_predictions.csv — per-variant predictions: AlphaMissense raw + Platt
• test_predictions_ensemble.csv — per-variant predictions: ensemble
• test_predictions_mlp.csv — per-variant predictions: MLP
• test_predictions_revel.csv — per-variant predictions: the M2+R model (Model Report 4 record — history, kept and labelled)
• test_predictions_trees.csv — per-variant predictions: tree ensembles
• headtohead_20261026_predictions.csv — per-variant predictions for the Model Report 6 head-to-head (allele_id + prob_A Model 2, prob_B M2+R, prob_C GBM+R — the adopted final model and this page's per-gene source — prob_D RF+R; labels and genes join from test_predictions.csv on allele_id)

Not for clinical use.

Product: Variant Evidence Agent · API: API documentation · Methods: Temporal evaluation · Calibration.