Evidence you can inspect.
We reproduced published benchmarks, tested new models and audited actual sequencing data. The work corrected reliability problems; it did not demonstrate a new benchmark leader or perfect biological accuracy.
Public workspace access is not enabled. These results describe the documented research snapshot and local implementation, not a completed prospective laboratory validation.
Pre-screen prediction
Genentech/assaybench · biogrid/yearfold0: 1,349 training, 218 validation and 334 public test screens. The benchmark ranks genes from released experimental context without using target measurements as scoring inputs. Published expert prompts can include post-hoc notes, significance criteria and ranking rationale; this comparison does not establish a clean pre-experiment historical forecast. The official evaluator uses adjusted nDCG@100.
| Method | AnDCG@100 |
|---|---|
| Oracle: best past screen, chosen with hindsight | 0.291784 |
| SplicR | 0.219952 |
| Gemini 3 + GPT-5.4 ensemble, same gene library | 0.219324 |
| Gemini 3 + GPT-5.4 ensemble, as published | 0.163091 |
| Phenotype prior, pre-2022 screens | 0.136052 |
On the same released context, without the target's measured gene library, SplicR's phenotype router scored 0.160369 against 0.163091 for the published frontier ensemble. Router minus ensemble: -0.002722; paired publication-bootstrap 95% interval [-0.015581, 0.006268] over 63 publications. The interval spans zero: no advantage was established and the candidate was not promoted.
Published model rankings were replayed from saved files. No fresh model API calls were made. The oracle selects with test answers and is not deployable. The public test had already been explored historically, so it is not an untouched prospective cohort.
Why earlier SplicR numbers differ
The original supplied-library phenotype prior reproduced at 0.136052. A separate archived fusion scored 0.219952 versus 0.219324 for an equally filtered/padded ensemble, with a paired interval spanning zero. Those experiments used the target's measured gene library and different training inputs. Their scores cannot be compared directly with the unaltered rankings above. Full details remain in the reproduction report.
Which hits reproduce
Given a screen a lab has already run, rank its candidates by how likely each is to be called a hit in an independent screen. This is a different task from the pre-screen prediction table and the two are not comparable. Measured 2026-09-28 on the SplicR cross-screen replication benchmark: 124 held-out screen pairs from different publications, first authors and libraries, same cell line, unperturbed proliferation. The model was frozen before this split was scored, and this split was scored once.
| Ranking | Average precision | Precision@10 |
|---|---|---|
| SplicR reliability ranking | 0.3538 | 0.915 |
| The screen's own effect size | 0.2464 | 0.696 |
Paired difference, resampled by screen pair: 0.107438, 95% interval [0.101975, 0.112786] over 124 pairs. On the 11 pairs outside the dominant library comparison the difference is 0.072126 [0.047362, 0.097934].
- Cross-screen replication is a proxy for reproducibility, not proof that a candidate passes an independent biological validation assay.
- The target screen's hit call depends on its own significance criteria; a real effect can fail to replicate because that threshold was stricter.
- 113 of the 124 held-out pairs are one library comparison (Behan 2019 against Meyers 2017); the 11 remaining pairs are reported separately.
- Unperturbed proliferation screens only. Nothing is established for drug-modifier, host-pathogen or reporter screens.
- This is a ranking, not a calibrated probability that any individual gene validates.
Later research results
Measured 2026-09-28 on the validation_2021 (218 screens, 34 publications) split. Development results on the 2021 validation split, not the 334-screen public test. Neither experiment established an improvement and neither model is promoted. These are development measurements, shown separately from the public-test table above because the two are not interchangeable.
| Experiment | Selected model | AnDCG@100 | 95% interval | Status |
|---|---|---|---|---|
| Gene-level LambdaRank over 57 historical-context features | E2_history_lambdarank | 0.180985 | [0.113378, 0.259514] | research only |
| Requested-effect polarity conditioning | P0_baseline_D3 | 0.099450 | [0.031147, 0.217461] | research only |
- Gene-level LambdaRank over 57 historical-context features
- Published frontier ensemble on the same 218 screens: 0.186437. Paired difference -0.005452, 95% publication-cluster interval [-0.016147, +0.010805] over 10,000 resamples. No improvement established: the interval contains zero and the point estimate is below the ensemble. Selection multiplicity 14.
- Requested-effect polarity conditioning
- Cross-validation selected the unconditioned baseline. The polarity prior scored -0.007931 [-0.023172, +0.004531] against it; the polarity-plus-transfer variant -0.010680 [-0.024030, +0.001069]. The parser separates all 47 metadata-matched increase/decrease comparisons that previous features collapsed. That is a representation improvement, not an established ranking improvement.
Neither experiment accessed the 334-screen public test, and neither model is promoted. The comparison table above is unchanged by this work.
Actual sequencing, compared with deposited counts
Four GSE145743 FASTQ files: 26,336,701 reads, with 65,383 guides compared per sample. The author table merges libraries A and B; the comparison uses library A counts per million (CPM). The deposited table is normalised to 10,000,000 reads per column, 5,000,000 per library across the two halves it merges, so counts are compared as counts per million rather than raw. Only GeCKOv2 library A guides are compared, because the deposited columns merge libraries A and B.
| Sample | Reads | Median CPM ratio | Spearman | Within 25% |
|---|---|---|---|---|
| plasmid | 3,119,503 | 0.9691 | 0.9289 | 89.49% |
| input | 12,449,815 | 0.9723 | 0.9399 | 88.85% |
| DMSO1 | 5,896,213 | 0.9763 | 0.9533 | 84.01% |
| Olaparib1 | 4,871,170 | 0.9762 | 0.9548 | 84.09% |
A median ratio near one checks scale. Correlation and tolerance describe guide-level agreement; neither proves perfect counts. The downloadable JSON includes mapping rates and dropout differences.
Post-screen analysis is a different task
MAGeCK and DrugZ analyzed 123,411 author-processed guide rows from GSE145743. Correcting selection across two directional FDR families changed q < 0.1 calls from 39 to 22. Native statistics remain inspectable.
CHD1L was not significant under this analysis. QC failed under the existing dropout/depth criteria. The input is author-processed and subsampled; these findings are not interchangeable with the earlier raw-read experiment. No gain in independent validation precision is established.
Source study and deposited data: GSE145743What the numbers mean
- AnDCG
- Ranking quality relative to the benchmark's ideal and random baselines. A score of 0.16 does not mean 16% accuracy.
- FDR
- A statistical error-rate measure for a selected set of discoveries under a method's assumptions. It is not one gene's validation probability.
- Independent validation
- New experimental outcomes withheld during development. No adequate cohort was established here, so validation-success percentages remain unavailable.
Download the evidence
These files are generated from the research records. Source-artifact hashes are included in the JSON. Technical reports refer to scripts and additional artifacts in the project repository; the downloads alone are not a complete computational environment.
- Machine-readable measurements
- SHA256 checksums for these downloads
- Benchmark reproduction and protocol
- Validation results and limitations
- Research results and implementation status
- How to read the evidence
- Post-screen methods and audit
- Website and project verification