Omevra · Benchmark
The Rare Disease Evidence Benchmark
One case is one family. The agent gets the sequencing data, the pedigree and the phenotype, and must find the causal variant and prove it, with a cited source for every ACMG code it applies.
In development · v0 not yet released
What a case contains
Inputs: reads or a joint VCF for the proband and parents (some cases add siblings), a PED file, and HPO phenotype terms for the proband.
Hidden truth: the causal variant or pair, the inheritance mode (de novo, recessive, compound het, X-linked), the expected ACMG codes, and the records that support each code.
Planted traps: calling artifacts that look pathogenic, such as a false positive in a low-complexity region, or a Mendelian violation caused by a dropout. Catching them is part of the score.
Every case is built from public data, so no patient data, IRB or HIPAA is involved. Families are synthetic: known pathogenic variants are spiked into real genome backgrounds, so the right answer is known by construction.
Layer 1 - Calling
Produce or check the calls from reads. Scored with hap.py precision, recall and F1 against the truth set, plus whether each planted trap was caught or missed.
Layer 2 - Family reasoning
Use the pedigree to filter and phase. Scored on the correct inheritance mode, and on correct de novo and compound het calls.
Layer 3 - Evidence
Apply ACMG codes, each one citing a record. Scored on code-level precision and recall. A cited record that does not support its claim counts as a fabrication, not as a near miss.
Layer 4 - Answer
Rank the causal variant, give a calibrated probability, or decline. Scored on top-1 and top-5 rank, calibration (ECE), and correct declines.
Two rules that keep it hard and honest
Temporal truth: the scoring set only uses labels that appeared after a fixed cutoff, and every agent declares its training cutoff.
Rolling rounds: each new ClinVar release adds fresh cases, scored blind. Older rounds become public practice sets, so the test cannot be memorised and stays hard as models improve.
Data sources
GIAB Ashkenazi trio (HG002, HG003, HG004, v4.2.1 benchmark) for real trio backgrounds and hap.py scoring. Platinum Pedigree (CEPH 1463) for multi-generation inheritance checks. ClinVar dated releases for variants to spike in and for temporal truth. ClinGen expert-panel (VCEP) curations for code-level ACMG truth. MaveDB functional scores for future blind rounds. HPO for phenotype terms.
Before any derived case is published, the license of each source is confirmed for redistribution. That check is an open item, and it is stated here rather than assumed.
Status
This benchmark is in development. v0 is not yet released, there is no leaderboard, and no model results are published yet, because none have been run. When the first cases and baseline runs exist, they will be published here in full, including the failures.
Rare disease labs and core facilities who want to try v0 as design partners can contact Omevra. Research Use Only.