# External validation at five or more sites in two countries before clearance

Source: https://onco.cc/ideas/idea-data-multisite-validation-precondition/  
OnCo record `idea-data-multisite-validation-precondition` (Idea). Data CC BY-NC 4.0, attribute "Data from OnCo (onco.cc)"; commercial use needs a licence.

## TL;DR

No cancer AI would be approved until it has been tested on patients from at least five different hospitals in at least two countries, none of which contributed training data.

## Summary

Cleared AI devices have frequently been validated on data from one or two sites, often overlapping with development sites. The proposal sets a minimum external validation requirement (at least five independent sites, at least two countries or health systems, no training-site overlap, pre-registered analysis, subgroup reporting) for regulatory clearance of cancer AI, with the sequestered benchmarks as one accepted route.

## Fields

- Kind: Idea
- Last checked: 2026-09-08
- Hypothesis: Models meeting the requirement will show smaller performance drops on deployment than models cleared under current rules, and the requirement will not materially slow clearance for well-built models.
- Rationale: Generalisation failure across sites is the best-documented failure mode of medical AI; multi-site external validation is the direct test and is inexpensive relative to the harm of deploying brittle models.
- Proposed test: Compare post-deployment performance drop for cleared models grouped by number of external validation sites; if the association holds, adopt the requirement and re-measure.
- Maturity: speculative
- Actor: regulator

## Sources

- Bottleneck evidence (AI that is built but not validated or deployed): Wu et al., How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals (Nature Medicine 2021): https://doi.org/10.1038/s41591-021-01312-x

## Connected records

- ideas: [Sequestered, prospectively collected benchmark datasets that no one can train on](https://onco.cc/ideas/idea-data-sequestered-prospective-benchmarks/)
- fronts: [AI & Computation](https://onco.cc/fronts/ai-computation/)
- bottlenecks: [AI that is built but not validated or deployed](https://onco.cc/bottlenecks/b-ai-validation/), [Regulatory divergence between regions](https://onco.cc/bottlenecks/b-regulatory-fragmentation/)
- key papers: [How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals](https://onco.cc/key-papers/paper-wu-nat-med/)
- terms: [External validation](https://onco.cc/terms/external-validation/)

---
JSON: https://onco.cc/api/v1/entities/idea-data-multisite-validation-precondition.json