# Mandatory subgroup performance reporting for cancer AI

Source: https://onco.cc/ideas/idea-data-subgroup-performance-reporting-mandate/  
OnCo record `idea-data-subgroup-performance-reporting-mandate` (Idea). Data CC BY-NC 4.0, attribute "Data from OnCo (onco.cc)"; commercial use needs a licence.

## TL;DR

Every AI tool would have to report how well it works for women and men, different ethnic groups, ages, scanner types and hospitals, not just an overall score.

## Summary

Cancer AI is often validated on populations that do not match deployment populations; performance gaps by skin tone (dermatology), breast density, ethnicity and scanner vendor are documented. The proposal requires, for clearance and in the model registry, performance reporting across a standard set of subgroups with minimum sample sizes and confidence intervals, and labelling restrictions where performance is unknown or inadequate.

## Fields

- Kind: Idea
- Last checked: 2026-09-08
- Hypothesis: Mandatory subgroup reporting will reveal clinically meaningful performance disparities in a substantial share of cleared cancer AI and lead to label restrictions or retraining for those models.
- Rationale: Pulse oximetry's racial bias went unrecognised for decades because subgroup performance was not required; AI will repeat this at scale unless reporting is mandatory.
- Proposed test: Evaluate ten cleared cancer AI devices on the standard subgroup set using sequestered data; publish disparities; track subsequent label changes.
- Maturity: speculative
- Actor: regulator

## Sources

- Bottleneck evidence (AI that is built but not validated or deployed): Wu et al., How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals (Nature Medicine 2021): https://doi.org/10.1038/s41591-021-01312-x

## Connected records

- fronts: [AI & Computation](https://onco.cc/fronts/ai-computation/)
- technologies: [Dermoscopy, total-body photography & AI skin analysis](https://onco.cc/technologies/dermoscopy-ai/), [Mammography & tomosynthesis](https://onco.cc/technologies/mammography/)
- bottlenecks: [AI that is built but not validated or deployed](https://onco.cc/bottlenecks/b-ai-validation/), [Trials do not represent the people who get cancer](https://onco.cc/bottlenecks/b-trial-diversity/)
- key papers: [How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals](https://onco.cc/key-papers/paper-wu-nat-med/)

---
JSON: https://onco.cc/api/v1/entities/idea-data-subgroup-performance-reporting-mandate.json