{"entity":{"id":"idea-data-multisite-validation-precondition","kind":"idea","name":"External validation at five or more sites in two countries before clearance","aka":[],"tldr":"No cancer AI would be approved until it has been tested on patients from at least five different hospitals in at least two countries, none of which contributed training data.","summary":"Cleared AI devices have frequently been validated on data from one or two sites, often overlapping with development sites. The proposal sets a minimum external validation requirement (at least five independent sites, at least two countries or health systems, no training-site overlap, pre-registered analysis, subgroup reporting) for regulatory clearance of cancer AI, with the sequestered benchmarks as one accepted route.","asOf":"2026-09-08","links":[{"label":"Bottleneck evidence (AI that is built but not validated or deployed): Wu et al., How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals (Nature Medicine 2021)","url":"https://doi.org/10.1038/s41591-021-01312-x"}],"tags":[],"related":["idea-data-sequestered-prospective-benchmarks"],"cancers":[],"sections":["ai-computation"],"technologies":[],"targets":[],"drugs":[],"companies":[],"institutions":[],"pathways":[],"terms":[],"trials":[],"people":[],"bottlenecks":["b-ai-validation","b-regulatory-fragmentation"],"keyPapers":["paper-wu-nat-med"],"journals":[],"dependsOn":[],"notes":[],"hypothesis":"Models meeting the requirement will show smaller performance drops on deployment than models cleared under current rules, and the requirement will not materially slow clearance for well-built models.","rationale":"Generalisation failure across sites is the best-documented failure mode of medical AI; multi-site external validation is the direct test and is inexpensive relative to the harm of deploying brittle models.","test":"Compare post-deployment performance drop for cleared models grouped by number of external validation sites; if the association holds, adopt the requirement and re-measure.","maturity":"speculative","actor":"regulator","cost":"small","horizonYears":2},"route":"/ideas/idea-data-multisite-validation-precondition/","neighbours":{"idea":[{"id":"idea-data-sequestered-prospective-benchmarks","kind":"idea","name":"Sequestered, prospectively collected benchmark datasets that no one can train on","route":"/ideas/idea-data-sequestered-prospective-benchmarks/"}],"section":[{"id":"ai-computation","kind":"section","name":"AI & Computation","route":"/fronts/ai-computation/"}],"bottleneck":[{"id":"b-ai-validation","kind":"bottleneck","name":"AI that is built but not validated or deployed","route":"/bottlenecks/b-ai-validation/"},{"id":"b-regulatory-fragmentation","kind":"bottleneck","name":"Regulatory divergence between regions","route":"/bottlenecks/b-regulatory-fragmentation/"}],"paper":[{"id":"paper-wu-nat-med","kind":"paper","name":"How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals","route":"/key-papers/paper-wu-nat-med/"}],"term":[{"id":"external-validation","kind":"term","name":"External validation","route":"/terms/external-validation/"}]}}