Open platform standardizes protein language model benchmarking and exposes homology leakage
- Protify is an open-source, low-code platform that puts zero-shot, supervised and transfer-learning evaluation of protein language models under one standardized pipeline.
- It was used on ProteinGym and a broad supervised panel covering enzymes, GO annotation, interactions and localization; no single model dominated, and gaps between leaders were within run-to-run noise.
- On enzyme optimal pH, a generic probe matched a specialized model, but under a homology-controlled split pooled-embedding probes lost nearly all signal, so rankings depend heavily on evaluation design.
💡 Points Based on the abstract
If the leaders sit inside seed noise, a leaderboard ordering is mostly a coin flip, so I'd want variance bars before trusting any rank. The pH result is the sharper warning: a probe that looks competitive on random splits but collapses once near-homologs are removed was probably reading sequence similarity. On my own target family, I'd rerun at a strict identity cutoff first.