Calibration

A calibrated probability means what it says: a “90% pathogenic” output should be right about 90% of the time. The project compares two calibration maps throughout — Platt scaling and isotonic regression — each fitted on train data only, never on the test set. Isotonic calibration is the map the final model ships: it fits a monotone step function from raw scores to observed frequencies on a train holdout, so the reported probabilities are the holdout's observed behaviour, not the model's own confidence.

Calibration quality is measured as expected calibration error (ECE): predictions are placed in 10 equal-width bins, and ECE is the bin-size-weighted average gap between mean predicted probability and observed fraction pathogenic, reported alongside AUROC and Brier. Discrimination and calibration are reported as separate claims for every model variant, including the ones that got worse.

The first calibration attempt made probabilities worse, and the record keeps it. Platt scaling fitted on the training window was applied to variants reclassified after the freeze: ECE moved from 0.040 (uncalibrated AlphaMissense) to 0.157, at unchanged AUROC 0.921 — Platt is a monotonic transform, so ranking survives while the probabilities stop meaning what they say. Adding the gene prior lifted AUROC to 0.959 while the raw ensemble's calibration worsened to ECE 0.251, the worst on the page. Isotonic calibration fitted on the train holdout brought the ensemble to ECE 0.017; the final model — GBM+R (gradient boosting + REVEL + isotonic), adopted 2026-10-26 in Model Report 6 — reports AUROC 0.970, ECE 0.025, Brier 0.062 on the 10,357 scored variants of the post-freeze test set.

Reliability — predicted against observed

Calibration error across the ablation

Calibration is reported per gene because an aggregate number can hide badly miscalibrated subgroups. For the final model (GBM+R), per-gene ECE is published for the 24 genes with n ≥ 50 scored test variants — and ordered worst first, so the failures lead:

23 of the 24 genes sit between 0.005 and 0.154. USH2A, at 0.204, is a calibration failure by the project's own standard and stays in the table, drawn in red on the same axis as every other gene. The adoption was not uniform per gene: USH2A improved from the M2+R model's 0.290, while MUTYH degraded 0.083 → 0.154 and NF1 moved 0.117 → 0.130 (Model Report 6).

Related method pages: Evaluation · Temporal evaluation. Product: Variant Evidence Agent.