OmevraOmevra · Variant Evidence Agent
Variant Evidence Agent architecture
Retrieval, scoring, calibration — and the gate, enforced in code, that decides whether a report may speak at all.
The flow, drawn
A report is produced by a pipeline in which the learned parts do one narrow job — assign a probability — and everything around that job is plain code with fixed behaviour. The served path is app/agent_service.py in the live application, calling the project's own vus_agent package; nothing in it reads pre-written report text.
The same rule at every stage: what the system may say is decided before any wording exists. The wording is the last step, and it is assembled from the evidence list — never the other way round.
Retrieval — what may count as evidence
The application loads the scored universe once at startup — the test-set feature table merged with the stored model predictions on allele id (the GBM+R probabilities since the 2026-10-26 adoption) (AgentService). A lookup resolves by allele id, or by gene symbol plus variant text appearing in the variant's name. For a resolved variant, the evidence is assembled from prediction-time records only — the frozen, Jan 2024 worldview:
- the AlphaMissense score and class, cited as
AM:chr:pos:ref>alt; - the frozen ClinVar status (the classification as it stood at the cutoff), cited as
CLINVAR:allele<id>; - the gene prior from the frozen train labels, cited as
GENEPRIOR:<gene>:frozen2024— present only when the variant also carries a predictor score.
The variant's later reclassification is never evidence — the service's own header states it in capitals. In the project's core package (vus_agent/agent.py), gather_evidence builds the same kind of list from a variant's features — AlphaMissense, REVEL, CADD, gnomAD allele frequency, gene constraint (pLI), and each ClinVar submission — one EvidenceItem (source, detail, record id) per fact. A missing feature is not an error: as vus_agent/features.py puts it, missing evidence is a first-class signal for the refusal path.
Scoring and calibration
Scoring happens in the project pipeline, not at request time: the probability a report carries is the stored model prediction for that allele (GBM+R — gradient boosting over AlphaMissense, the gene prior and REVEL, isotonic-calibrated — since 2026-10-26), produced by the pipeline described on the product page and joined to the variant when the service starts. Calibration is likewise precomputed and mechanical — vus_agent/calibrate.py holds the maps (Platt scaling by gradient descent; isotonic regression by the pool-adjacent-violators algorithm) and the metrics (ECE in 10 bins, Brier, rank-based AUROC). The standing rule — maps fitted on train data only, error reported per gene — is on the calibration method page; the split that makes those numbers mean something is on the temporal evaluation page.
The gate — which decisions are code's, which are the model's
The model decides one thing
The probability. The model's output — passed through the calibration map — is a ranking of pathogenic likelihood for variants the predictors cover. It does not decide whether a report may be written, what the report may claim, or how a refusal is phrased.
Code decides whether anything may be said
assess (and its served twin assess_real) applies a constant: MIN_EVIDENCE = 3. Fewer than 3 distinct evidence items — or no probability — and the result is the decline: verdict “insufficient evidence — remains VUS”, probability null, and the citations still list whatever was retrieved. The decline text is a fixed string in the code, not generated wording, so a refusal can never be talked out of its shape.
Code decides what a claim is
Every evidence line in a report is built from an EvidenceItem and carries its record id in angle brackets; the report's citation list is those ids and nothing else. There is no path by which a claim enters a report without the record it rests on — and the project's adversarial test (tests/test_agent_refuses.py) exists to prove the refusal side: a fabricated citation or a missing record must end in a decline.
The optional synthesis layer — fenced, and off by default
If, and only if, the server environment carries an API key, a summary writer may re-word evidence the system has already assembled into short prose. Its instruction is a fence: use only the supplied evidence, preserve every citation identifier verbatim, add no facts, scores, classifications or interpretations, state no probability that is not in the evidence — and if the evidence is a refusal, say exactly that and nothing more. It runs at temperature 0 with a 120-word cap; on any failure it returns nothing and the deterministic report stands on its own. Without the key the module reports itself disabled and the interface never mentions it.
Live-evidence mode — retrieval only, no score
For a variant outside the scored universe, the application does not extrapolate. app/live_evidence.py retrieves current public records and returns them as retrieved: ClinVar, via NCBI E-utilities (esearch, then esummary — the top 5 records, with title, current classification and review status), and gnomAD, via its public GraphQL API (dataset gnomad_r4) — which needs GRCh38 coordinates; without them that half reports itself “unavailable”, with the reason. The panel is labelled “Live retrieved evidence”, with its note stated in the payload itself: no model score is attached to this panel. Every failure is a status — ok, not_found, failed, unavailable — with the reason attached; content is never invented to fill a gap. The two sources are called with fixed timeouts (12 s for ClinVar, 45 s for the slower gnomAD API).
The batch flow
A batch (app/batch.py) is a pasted list — one variant per line or comma-separated — or a single gene name, which expands to every variant of that gene in the scored universe. Entries are parsed (gene + variant text, a full transcript-style name, an allele id, or a bare token), de-duplicated, and bounded: input text is capped at 20,000 characters, and a batch resolving to more than 25 variants is rejected with a clear message — never truncated. Each variant then runs through exactly the code path the single-variant flow uses: in the universe, the agent report (summarised or declined); outside it, live-evidence mode. External calls are serialised behind a lock with at least 0.5 s between them, so a batch never fires parallel requests at the public sources. The output counts the three outcomes separately — summarised, declined, live-evidence — and ships two packs (a Markdown document and a JSON file) whose per-variant reports are the individual reports, unchanged.
Scope
A research system, not a certified medical product, and not for clinical use: a calibrated probability is not a clinical classification. Product and results: Variant Evidence Agent · Methods: Calibration · Temporal evaluation.
This page describes vus_agent/agent.py, vus_agent/calibrate.py and vus_agent/features.py in the project, and app/agent_service.py, app/live_evidence.py, app/batch.py and app/synthesis.py in the live application — the code as it stands.