# A public benchmark and audit of chatbot answers to cancer questions

Source: https://onco.cc/ideas/idea-moon-ai-cancer-answer-audit/  
OnCo record `idea-moon-ai-cancer-answer-audit` (Idea). Data CC BY-NC 4.0, attribute "Data from OnCo (onco.cc)"; commercial use needs a licence.

## TL;DR

Patients now ask AI assistants about their cancer. Test those assistants regularly on real questions, publish the scores, and certify the ones that meet the bar.

## Summary

Large language models are becoming a primary source of medical information; studies show variable accuracy, occasional dangerous advice and inconsistent sourcing on oncology questions. The proposal is an independent, continuously updated benchmark of patient-style cancer questions (treatment options, side-effects, alternative therapies, prognosis) scored by oncologists and patients for accuracy, safety, sourcing and readability, with public leaderboards and a certification mark for assistants that meet thresholds and disclose limitations.

## Fields

- Kind: Idea
- Last checked: 2026-09-08
- Hypothesis: Public auditing raises the accuracy and safety of assistant answers to cancer questions across vendors within a year, and certified assistants are preferred by patient organisations.
- Rationale: Benchmarks drive model behaviour in AI development; making the oncology benchmark public and patient-facing turns that pressure toward safety.
- Proposed test: Run the benchmark quarterly on major assistants for a year; measure score trajectories and adoption of the certification by patient organisations and health systems.
- Maturity: early-clinical
- Actor: data

## Sources

- Bottleneck evidence (Misinformation and unproven therapies): Johnson et al., Cancer misinformation and harmful information on Facebook and other social media (JNCI 2022): https://doi.org/10.1093/jnci/djab141

## Connected records

- bottlenecks: [AI that is built but not validated or deployed](https://onco.cc/bottlenecks/b-ai-validation/), [Misinformation and unproven therapies](https://onco.cc/bottlenecks/b-misinformation/)
- key papers: [Cancer Misinformation and Harmful Information on Facebook and Other Social Media: A Brief Report](https://onco.cc/key-papers/paper-johnson-j-natl-cancer-inst/)

---
JSON: https://onco.cc/api/v1/entities/idea-moon-ai-cancer-answer-audit.json