hybrid-perturb-bench

The benchmark runs on Norman et al. 2019 Perturb-seq (CRISPRa, K562), as harmonized by scPerturb: 111,445 cells × 33,694 genes, 11,855 control cells, 237 perturbation labels (control + 105 singles + 131 combinations). Profiles are pseudobulk counts per perturbation, library-normalized to 1e4 + log1p after aggregation. The evaluation gene set is the top 2000 genes by pseudobulk-profile variance across control + train perturbations only.

The split holds out entire perturbations: 21 test singles against 84 train singles (seed 0, 20%). The 131 combinations were deferred in milestone 1 and became the compositional-generalization test in milestone 2. The evaluation contract scores predicted against real pseudobulk profiles per perturbation — mse_delta / rmse_delta, pearson_delta, mse_expr / pearson_expr, and deg_overlap_at_50 (the share of the top-50 |Δ| genes recovered) — with a split-half ceiling: one half of a perturbation's cells scored against the other, Spearman-Brown corrected.

Three honest baselines under the contract, mean over the 21 held-out singles on the 2000-gene set:

The no-change baseline predicts a delta of exactly 0, so its Pearson correlation is undefined (zero variance) — it is ranked on MSE / overlap. Ridge covers 19 of 21 test perturbations: two targets are absent from the filtered expression matrix, so no co-expression features exist for them. Reading, in the project's own terms: the global mean clearly beats no-change, and the linear ridge does not beat the global mean on held-out perturbations — with control-only co-expression features it essentially collapses to the mean. Both sit far below the real-data ceiling — the headroom a real model has to earn.

The hybrid predicts the pseudobulk delta on the 2000 eval genes, parameterized as coefficients in a PCA basis of the 84 train deltas (40 components) around the train mean delta. Per perturbed gene it takes four inputs: gene embeddings (ESM-2, mean-pooled over residues; reviewed human UniProt sequences, coverage 105/105 genes), a co-expression block (the target gene's co-expression profile in control cells only), statistical / DEG-frequency features, and a pseudobulk-delta retrieval channel (a similarity-weighted mean of train perturbations' real deltas) — feeding an MLP (input → 128 → 128 → 40, dropout 0.25, Adam, 500 epochs, seed 0). All fitted statistics use the 84 train singles only; all hyperparameters were fixed a priori and nothing was tuned on test. The retrieval channel alone is also scored as the retrieval_knn reference predictor.

On held-out single perturbations the hybrid does not beat the baselines it was built to challenge. It improves on no-change (MSE 0.005134 vs 0.006534) and on its own retrieval channel (0.5366 vs 0.5076 Pearson), but it sits below the global mean (0.5876) and the ridge (0.5785) on Pearson Δ, and below both on DEG overlap (0.3143 vs 0.3314 / 0.3326). With 84 training perturbations, sequence and co-expression features do not identify what an unseen single perturbation will do beyond the strong shared CRISPRa response program that the global mean already captures. This negative result is reported as-is; no hyperparameter was adjusted after seeing it.

The 131 two-gene combinations were never used in any training. additive_hybrid is the sum of the hybrid's two predicted single deltas; the hybrid (direct) averages the two component feature blocks.

On combinations the picture reverses: both hybrid variants score far above the global mean and the ridge (Pearson 0.7995–0.8248 vs 0.5663–0.5753), though still below the real-data ceiling (0.9314 Pearson, 0.7011 DEG overlap). Subgroups by whether the combo's components are train or test singles:

In-sample flag: additive_hybrid's both-train subgroup benefits from the hybrid having been trained on those singles — its single predictions there are partly in-sample — so the mixed and both-test subgroups are the cleaner generalization read, and the hybrid's advantage over the baselines persists there. Combo deltas are also larger in magnitude than single deltas, and many Norman combos are dominated by one strong component, which makes per-combo correlation an easier target than for singles.

GEARS was attempted as a comparator and not completed: the run stopped inside GEARS' per-cell graph construction, before training began, after roughly 43 minutes of processing on this machine with memory projected to exceed what was available. No GEARS predictions exist and no GEARS numbers are reported anywhere in the project's results. Nothing in this benchmark should be read as a statement about GEARS' accuracy.

• The hybrid does not beat the global mean or the ridge on held-out single perturbations — the headline singles claim remains unmet, and the benchmark's “beat the mean” bar is still open there.
• No GEARS comparison exists; nothing here is a statement about GEARS' accuracy.
• One dataset, one cell line (Norman 2019, K562): no cross-cell-line, cross-dataset, or zero-shot claim is made.
• The hybrid's design and hyperparameters were fixed before its test scores were seen and were not revised afterward.

Every figure on this page is taken from the project's RESULTS.md (milestones 1 and 2) and is recomputed against that file at build time.

The standalone proof page for this product: Evaluations — the protocol, the versioned results, and every result file as a download.