OmevraOmevra · Validation
Validated on what came next
Most variant tools are tested on data they were tuned on. Omevra’s headline numbers come from a temporal split: train on the past, then test on what the field decided afterwards.
The temporal split
The model was fitted on ClinVar as frozen in January 2024, and tested on variants whose classifications were made or changed after that freeze — a test set the model never trained on.
A random split leaks: the same variant, the same gene, and the same submitter patterns appear on both sides, so the score flatters the model. A temporal split cannot leak that way. It asks the only question a laboratory actually cares about — does the model hold up on the variants nobody had classified yet? The full method is on the temporal evaluation page, and the complete per-model results are in the evaluations.
The declined 62.7% is a coverage limit, stated as one: the model scores the missense variants that carry predictor scores, and declines the rest rather than inventing a probability. Extending coverage — splice and non-coding variants — is next on the roadmap.
Calibration, and cite-or-decline
A probability is only useful if it means what it says. Calibration is reported alongside discrimination on every result Omevra publishes.
ECE (expected calibration error) measures the gap between stated probabilities and observed outcomes: when the model says 0.80, is it right about 80% of the time? The final model reaches AUROC 0.970 with ECE 0.025 on the scored variants. The method, including per-gene calibration, is documented on the calibration page.
Every claim in an evidence report cites the record retrieved for it. Where the evidence is too thin to support a claim, the system declines — and the decline is shown as a decline, in the report and in the coverage numbers above, never dressed up as a result.
The fallback model
The current model uses REVEL scores among its inputs. An earlier logistic model that uses no REVEL at all scored AUROC 0.958 on the same temporal test set. That model is kept as the fallback: if REVEL’s commercial licence cannot be confirmed before launch, the platform ships on the no-REVEL model instead. See the data-licence note below.
Web Agent, measured the same way
The research systems are held to the same standard. In its final evaluation, the Web Agent completed 36 of 36 tasks with its guardrails on (95% CI 90.3–100%) and 27 of 36 with them off (95% CI 57.8–87.9%). On the adversarial suite it scored 9 of 9 (95% CI 66.4–100%) against 0 of 9 (95% CI 0–33.6%) with guardrails off.
The rest of the record
Hybrid Perturb-Bench
The hybrid singles model scored 0.537, below the global-mean baseline of 0.588, against a measured ceiling of 0.917. Below baseline — published as-is, unchanged.
NGS Pipeline Atlas
33 real sequencing runs processed end to end. A second full run came out byte-identical — MD5 822aad367e679cf720aa7ab3398dda89 — so the pipeline’s results can be re-checked instead of merely believed.
Multi-Agent Evidence
Measured head to head against a single agent, both systems scored 1.00 on citation accuracy — the ceiling of that benchmark, so the test cannot separate them — and the multi-agent system cost more to get there. A harder benchmark is needed.
Data licences
AlphaMissense predictions © 2023 DeepMind Technologies Limited, used under CC BY 4.0 — commercial use permitted with attribution. ClinVar is public domain. gnomAD is publicly available. REVEL scores are distributed under academic terms; confirming the commercial licence is a pre-launch step, and the platform’s no-REVEL model (AUROC 0.958) is the fallback.