OmevraOmevra · hybrid-perturb-bench — proof
hybrid-perturb-bench — evaluations
Milestones 1 and 2 under one contract — combinations, held-out singles, and the metric files.
Evaluation — Norman 2019 Perturb-seq
Version labels: Milestone 1 (baselines) and Milestone 2 (hybrid), same contract. Norman et al. 2019 Perturb-seq (CRISPRa, K562), as harmonized by scPerturb: 111,445 cells × 33,694 genes, 11,855 control cells. Profiles are pseudobulk counts per perturbation, library-normalized to 1e4 + log1p. The split holds out entire perturbations: 84 train singles, 21 held-out test singles (seed 0); the 131 two-gene combinations were never used in any training and form the compositional-generalization test. The contract scores predicted against real pseudobulk profiles on the top 2000 eval genes: Pearson Δ (pearson_delta), MSE Δ, and DEG overlap @50 — the share of the top-50 |Δ| genes recovered (deg_overlap_at_k in the files, k = 50). The ceiling is a split-half of real data, Spearman-Brown corrected.
Combinations — Milestone 2 (n = 131)
| Method | n | Pearson Δ | MSE Δ | DEG overlap @50 |
|---|---|---|---|---|
| ceiling (split-half, real data) | 131 | 0.9314 | 0.004406 | 0.7011 |
| additive_hybrid (sum of predicted single deltas) | 131 | 0.8248 | 0.005471 | 0.5472 |
| hybrid (direct) | 131 | 0.7995 | 0.008824 | 0.5098 |
| linear ridge | 129 | 0.5753 | 0.012948 | 0.3033 |
| global perturbation mean | 131 | 0.5663 | 0.013006 | 0.2921 |
Pearson Δ: additive hybrid 0.8248, hybrid 0.7995, ridge 0.5753, global mean 0.5663, ceiling 0.9314. DEG @50: additive hybrid 0.5472, hybrid 0.5098, ridge 0.3033, global mean 0.2921, ceiling 0.7011. In-sample flag: additive_hybrid's both-train subgroup (Pearson 0.8698) is partly in-sample; the mixed (0.7744) and both-test (0.7443) subgroups are the cleaner read, and the advantage persists there.
Held-out singles — Milestone 2 (n = 21) — published as-is
| Method | n | Pearson Δ | MSE Δ | DEG overlap @50 |
|---|---|---|---|---|
| ceiling (split-half, real data) | 21 | 0.9167 | 0.001382 | 0.6590 |
| global perturbation mean | 21 | 0.5876 | 0.004324 | 0.3314 |
| linear ridge (co-expression features) | 19 | 0.5785 | 0.004470 | 0.3326 |
| hybrid (ESM-2 + co-expression + stats + retrieval → MLP) | 21 | 0.5366 | 0.005134 | 0.3143 |
| retrieval_knn (retrieval channel alone) | 21 | 0.5076 | 0.005692 | 0.2762 |
| control mean (no-change) | 21 | NaN | 0.006534 | 0.0210 |
The hybrid is below the baselines on singles, and that stays published. Hybrid Pearson Δ 0.5366 sits below the global mean 0.5876 and the ridge 0.5785 (ceiling 0.9167); DEG @50 tells the same story — hybrid 0.3143 against global mean 0.3314 and ridge 0.3326 (ceiling 0.6590). No hyperparameter was adjusted after seeing this result. GEARS was attempted as a comparator and not completed — no GEARS predictions exist and no GEARS numbers are reported.
Downloads — the metric files
- RESULTS.md — the full results document (Milestones 1 and 2)
- milestone1_metrics_aggregate.csv — Milestone 1 aggregate metrics
- milestone1_metrics_per_perturbation.csv — Milestone 1 per-perturbation metrics
- milestone2_metrics_aggregate.csv — Milestone 2 aggregate metrics (singles + combinations)
- milestone2_metrics_per_perturbation.csv — Milestone 2 per-perturbation metrics
- milestone2_run.json — Milestone 2 run record
- split.json — the train/test split
Each file is copied byte-identical from the project's results/ at build time; the build verifies the copies.
Product: hybrid-perturb-bench.