# A neutral public evaluator for cancer AI, on the model of NIST

Source: https://onco.cc/ideas/idea-data-neutral-ai-evaluator/  
OnCo record `idea-data-neutral-ai-evaluator` (Idea). Data CC BY-NC 4.0, attribute "Data from OnCo (onco.cc)"; commercial use needs a licence.

## TL;DR

Create an independent public body whose job is to test cancer AI tools against each other on locked-away data and publish the scores, so hospitals can buy on evidence.

## Summary

Hospitals cannot compare AI vendors; each presents its own validation. A publicly funded evaluator, running the sequestered benchmarks, publishing head-to-head results, subgroup performance and robustness tests, and updating as models change, would make procurement evidence-based and give regulators an independent data source. Models exist in NIST's testing programmes and the UK's AI evaluation initiatives.

## Fields

- Kind: Idea
- Last checked: 2026-09-08
- Hypothesis: Publication of independent head-to-head results will shift procurement toward better-performing models and cause under-performing products to leave the market within three years.
- Rationale: Independent testing works where buyers cannot verify claims themselves (cars, appliances, biometrics); cancer AI has exactly this information asymmetry.
- Proposed test: Fund the evaluator to test one task (mammography AI) across all vendors; survey procurement decisions in the following two years for reference to the results.
- Maturity: speculative
- Actor: policy

## Sources

- Bottleneck evidence (AI that is built but not validated or deployed): Wu et al., How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals (Nature Medicine 2021): https://doi.org/10.1038/s41591-021-01312-x

## Connected records

- ideas: [Sequestered, prospectively collected benchmark datasets that no one can train on](https://onco.cc/ideas/idea-data-sequestered-prospective-benchmarks/)
- fronts: [AI & Computation](https://onco.cc/fronts/ai-computation/)
- bottlenecks: [AI that is built but not validated or deployed](https://onco.cc/bottlenecks/b-ai-validation/)
- key papers: [How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals](https://onco.cc/key-papers/paper-wu-nat-med/)

---
JSON: https://onco.cc/api/v1/entities/idea-data-neutral-ai-evaluator.json