Why cross-split AUPRC gaps can mislead when judging deep learning antibiotic screens
- A methodological commentary on evaluating deep learning models for antibiotic screening, where only a handful of compounds can be followed up.
- Because AUPRC shifts with how common actives are in the test set, and molecular splitting changes both structural similarity and class balance, a gap between splits cannot by itself show how much structure drove the change.
- The authors advise reporting test-set composition and split method, comparing models on one fixed independent set, and judging utility by top-ranked performance, early enrichment or prospective hit rates.
💡 Points Based on the abstract
Anyone who has watched a scaffold-split score drop knows the temptation to call it "generalization loss". Here the baseline moves too, since random AUPRC equals prevalence. A useful companion is a prevalence-matched or normalized comparison, plus hit rate at the few picks your assay budget allows. It is a commentary, so it offers no new benchmark.
📄 Paper · Journal of Chemical Information and Modeling (ACS) ↗ Interpretive Limits of Cross-Split AUPRC Differences in Deep Learning-Based Antibiotic Screening