Healthy-looking diagnostics can hide missing states in generative samplers
- Shows analytically that self-normalized diagnostics such as effective sample size cannot detect basins a learned sampler never visits, so free energies are biased while confidence intervals look tight.
- Tested on a toy double well, exactly enumerated spin-glass instances and an alanine dipeptide Boltzmann generator, where masked diffusion and GFlowNet intervals often missed the truth.
- A reverse check scored against an independent reference sampler caught the omitted states, and peptide and miniprotein comparisons suggest such omissions can matter for lead optimization.
💡 Points Based on the abstract
ESS and bootstrap intervals only measure agreement among samples the model actually draws, so a tight interval speaks to precision, never to coverage. It is the same trap as an MD run stuck in one rotamer with a beautiful block-average error bar. On our own project, would we notice a missing cryptic-pocket or alternative-pose basin without an orthogonal reference such as replica exchange?