# A registry of external validation datasets for cancer AI models, with mandatory reporting

Source: https://onco.cc/ideas/idea-tr2-ai-external-validation-registry/  
OnCo record `idea-tr2-ai-external-validation-registry` (Idea). Data CC BY-NC 4.0, attribute "Data from OnCo (onco.cc)"; commercial use needs a licence.

## TL;DR

Cancer AI models are usually tested on data from the same hospital they were built on. A registry of independent test datasets, and a rule that every model reports performance on at least one, would show which models really work.

## Summary

Most published cancer AI models lack external validation, and performance drops sharply on data from other institutions. A curated registry of held-out datasets across modalities (pathology, radiology, genomics) hosted by neutral custodians, with a submission protocol that returns performance metrics without releasing the data, would make external validation routine. Journals and regulators would require a registry validation for any clinical claim.

## Fields

- Kind: Idea
- Last checked: 2026-09-08
- Hypothesis: Models validated through the registry will show a median performance drop of at least ten percentage points from internal to external validation, and the requirement will improve the external performance of subsequently published models.
- Rationale: Held-out evaluation servers (as in machine learning benchmarks) prevent overfitting to the test set; medicine has the datasets but not the shared infrastructure.
- Proposed test: Establish registry datasets for three tasks (HER2 scoring, lung nodule malignancy, ctDNA variant calling); validate 50 published models; report the distribution of performance changes.
- Maturity: early-clinical
- Actor: data

## Sources

- Bottleneck evidence (Preclinical results do not reproduce): Errington et al., Investigating the replicability of preclinical cancer biology (eLife 2021): https://doi.org/10.7554/eLife.71601

## Connected records

- ideas: [Patient-level multimodal foundation models for treatment selection](https://onco.cc/ideas/idea-multimodal-foundation-model/), [Public gold-standard datasets for validating every cancer biomarker test](https://onco.cc/ideas/idea-tr2-open-cdx-validation-sets/)
- technologies: [Digital pathology & AI](https://onco.cc/technologies/digital-pathology-ai/), [Pathology & radiology foundation models](https://onco.cc/technologies/pathology-foundation-model/)
- bottlenecks: [AI that is built but not validated or deployed](https://onco.cc/bottlenecks/b-ai-validation/), [Preclinical results do not reproduce](https://onco.cc/bottlenecks/b-reproducibility/)
- roadmaps: [AI in oncology roadmap: pattern readers → foundation models → agents in the workflow](https://onco.cc/roadmaps/ai-oncology-roadmap/)
- terms: [External validation](https://onco.cc/terms/external-validation/)

---
JSON: https://onco.cc/api/v1/entities/idea-tr2-ai-external-validation-registry.json