{"entity":{"id":"idea-data-drift-monitoring-standard","kind":"idea","name":"A standard for monitoring AI performance drift with pause thresholds","aka":[],"tldr":"Set common rules for how hospitals check that an AI tool still works as the scanners, patients and practices around it change, and when it must be switched off.","summary":"Model performance shifts when scanners, staining protocols, populations or clinical practice change. Few deployments monitor this. The proposal is a technical standard: a per-site reference dataset re-scored monthly, input distribution monitoring, calibration and subgroup checks, pre-specified thresholds for alert and pause, and a documented recalibration or retraining pathway, integrated with the vendor's change control plan and reported to the registry.","asOf":"2026-09-08","links":[{"label":"Bottleneck evidence (AI that is built but not validated or deployed): Wu et al., How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals (Nature Medicine 2021)","url":"https://doi.org/10.1038/s41591-021-01312-x"}],"tags":[],"related":["idea-data-ai-post-market-performance-reporting"],"cancers":[],"sections":["ai-computation"],"technologies":[],"targets":[],"drugs":[],"companies":[],"institutions":[],"pathways":[],"terms":[],"trials":[],"people":[],"bottlenecks":["b-ai-validation"],"keyPapers":["paper-wu-nat-med"],"journals":[],"dependsOn":[],"notes":[],"hypothesis":"Sites following the drift standard will detect degradation months earlier than unmonitored sites and avoid patient harm events attributable to silent drift.","rationale":"Industrial machine learning monitors drift as routine engineering practice; healthcare deployments largely do not, and the few audits done have found drift within a year or two of deployment.","test":"Implement the standard at ten sites running the same pathology or radiology model; compare detected drift events and time to detection with ten unmonitored sites over two years.","maturity":"early-clinical","actor":"engineering","cost":"small","horizonYears":2},"route":"/ideas/idea-data-drift-monitoring-standard/","neighbours":{"idea":[{"id":"idea-data-silent-trial-before-deployment","kind":"idea","name":"A mandatory silent (shadow) trial before any cancer AI goes live","route":"/ideas/idea-data-silent-trial-before-deployment/"},{"id":"idea-data-ai-post-market-performance-reporting","kind":"idea","name":"Mandatory post-market performance reporting for cancer AI","route":"/ideas/idea-data-ai-post-market-performance-reporting/"},{"id":"idea-data-ai-decommissioning-rules","kind":"idea","name":"Rules for retiring cancer AI when performance drops or the standard of care moves","route":"/ideas/idea-data-ai-decommissioning-rules/"}],"section":[{"id":"ai-computation","kind":"section","name":"AI & Computation","route":"/fronts/ai-computation/"}],"bottleneck":[{"id":"b-ai-validation","kind":"bottleneck","name":"AI that is built but not validated or deployed","route":"/bottlenecks/b-ai-validation/"}],"paper":[{"id":"paper-wu-nat-med","kind":"paper","name":"How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals","route":"/key-papers/paper-wu-nat-med/"}]}}