OmevraOmevra · Methods
Temporal evaluation
Freeze the classifications at a cutoff; test only on what was reclassified after it.
The freeze/test design
All data is open and pinned by date, because the temporal split depends on it. Classifications are frozen at the Jan 2024 cutoff. The train set is variants labelled at the cutoff and frozen; the test set is only variants that were uncertain at the cutoff and have since been reclassified — no test label existed at train time. Calibration maps are likewise fitted on train data only and judged against the untouched post-freeze test set.
Source: data/processed_v2/SPLIT_REPORT.md and docs/validation-dossier.md §2, §5. The cutoff snapshot held 2,366,650 GRCh38 rows; the current snapshot holds 4,578,190. Train: 1,103,657 variants (240,752 pathogenic, 862,905 benign). Test: 27,748 variants (8,856 pathogenic, 18,892 benign).
Why random splits leak
A variant reclassified after the cutoff was often reclassified using the same predictor scores and submissions that appear as features. Random splits put that variant — or its near-duplicates — in both train and test: the model is graded on evidence it has already seen. The design rule, written in the leakage audit before any model code: pin a dated release, train on classifications frozen at the cutoff, test only on variants whose classification changed after it, and join features as of the cutoff where the source allows dating — and say so per feature where it does not.
Near-duplicate guard
Test rows sharing gene + name with train are removed — the guard removed 0 rows in this build. The audit states the rule behind it: same variant / same codon across splits goes to train only.
The caveat, stated in the split report itself
Post-cutoff reclassifications are themselves partly predictor-influenced. The temporal split removes the direct leak; it does not make labels pure. Measured performance is therefore performance against real-world reclassification practice, not against predictor-free ground truth.
What the design makes visible
The Platt calibration failure only appears because the test set is temporal: a calibration map that does not transfer forward in time can still look fine on a random split. On this test set the failure, and then the isotonic improvement, are both visible — see Calibration.
Of the 27,748 post-freeze reclassifications, the agent summarises 10,357 (37.3%) with cited evidence and declines 17,391 (62.7%) — a decline is a designed output, not an error.
Related method pages: Evaluation · Calibration. Product: Variant Evidence Agent.
Architecture
How the systems are built
web-agent architecture — the state machine, grounding rules, budgets and the public-page rails, from the code. Variant Evidence Agent architecture — retrieval, scoring, calibration, and the cite-or-decline gate enforced in code.