{"entity":{"id":"b-reproducibility","kind":"bottleneck","name":"Preclinical results do not reproduce","aka":[],"tldr":"Fewer than half of landmark cancer biology findings reproduce when someone else tries.","summary":"Systematic attempts to replicate published cancer biology have failed more often than they have succeeded. Amgen scientists could confirm only 6 of 53 landmark findings; the Reproducibility Project: Cancer Biology found that effect sizes in replications were on average 85% smaller than in the original papers and that fewer than half of the effects replicated on strict criteria, while most original papers lacked the information needed to attempt a replication at all. Misidentified and contaminated cell lines contaminate tens of thousands of publications. The estimated cost of irreproducible preclinical research in the US is tens of billions of dollars a year, and the human cost is drugs taken into patients on foundations that do not hold. The causes are incentives for novelty over rigour, small underpowered experiments, flexible analysis, lack of blinding and randomisation in animal work, and no funding or credit for replication.","asOf":"2026-09-08","links":[{"label":"Errington et al., Investigating the replicability of preclinical cancer biology (eLife 2021)","url":"https://doi.org/10.7554/eLife.71601"},{"label":"Begley & Ellis, Raise standards for preclinical cancer research (Nature 2012)","url":"https://doi.org/10.1038/483531a"},{"label":"Freedman, Cockburn & Simcoe, The economics of reproducibility in preclinical research (PLOS Biology 2015)","url":"https://doi.org/10.1371/journal.pbio.1002165"},{"label":"NIH Rigor and Reproducibility","url":"https://grants.nih.gov/policy-and-compliance/policy-topics/reproducibility"}],"tags":[],"related":["depmap","hpa","cancer-models","pubmed-europepmc","idea-tr2-timestamped-eln","idea-tr2-tenure-replication-credit","idea-tr2-random-audit","idea-tr2-paid-statistical-review","idea-tr2-reference-model-panels","idea-tr2-ai-external-validation-registry","idea-tr2-two-lab-rule","idea-tr2-preclinical-living-reviews","idea-tr2-phd-replication-year","idea-moon-independent-replication-institute"],"cancers":[],"sections":["drug-discovery"],"technologies":["organoids","pdx-models","crispr-screens","functional-drug-testing"],"targets":[],"drugs":[],"companies":[],"institutions":["broad-institute","cold-spring-harbor","aacr","nci"],"pathways":[],"terms":[],"trials":[],"people":[],"bottlenecks":[],"keyPapers":["paper-begley-nature","paper-freedman-plos-biol"],"journals":[],"dependsOn":[],"notes":[],"stage":"data-knowledge","severity":"major","metrics":[{"label":"Landmark preclinical cancer findings reproduced by Amgen","value":"6 of 53 (11%)","source":"Begley & Ellis, Nature 2012","url":"https://doi.org/10.1038/483531a"},{"label":"Median reduction in effect size in replications vs original experiments (Reproducibility Project: Cancer Biology)","value":"85% smaller","source":"Errington et al., eLife 2021","url":"https://doi.org/10.7554/eLife.71601"},{"label":"Estimated annual US spend on irreproducible preclinical research","value":"~US$28 billion","source":"Freedman, Cockburn & Simcoe, PLOS Biology 2015","url":"https://doi.org/10.1371/journal.pbio.1002165"},{"label":"Published articles using misidentified or contaminated cell lines","value":"More than 32,000","source":"Horbach & Halffman, PLOS ONE 2017","url":"https://doi.org/10.1371/journal.pone.0186281"}],"causes":["Careers and grants reward novel positive results, not confirmation or null results.","Experiments are underpowered and analysed flexibly until a significant result appears.","Cell lines are misidentified or contaminated and reagents such as antibodies are poorly validated.","Animal studies are rarely randomised, blinded or pre-registered.","Methods sections omit the detail needed to repeat the work, and raw data are not shared."],"currentEfforts":["The Reproducibility Project: Cancer Biology (Center for Open Science and Science Exchange) published open replication attempts of 50 high-impact papers.","NIH rigour and reproducibility policy (2016) requires authentication of key biological resources including cell lines in grant applications.","The ARRIVE 2.0 guidelines and journal checklists set reporting standards for animal experiments.","Registered Reports at more than 300 journals commit to publication before results are known.","DepMap and the Human Protein Atlas provide systematic, reproducible baseline datasets that individual findings can be checked against.","ICLAC (International Cell Line Authentication Committee) maintains the register of misidentified cell lines and STR authentication standards."],"successLooksLike":"A majority of high-impact preclinical cancer findings replicate when tested independently, replication is funded and credited as first-class science, and no drug enters human trials on the basis of a single unreplicated preclinical result."},"route":"/bottlenecks/b-reproducibility/","neighbours":{"collection":[{"id":"cancer-models","kind":"collection","name":"Cancer Models (PDCM Finder) & HCMI","route":"/collections/cancer-models/"},{"id":"depmap","kind":"collection","name":"DepMap (Cancer Dependency Map)","route":"/collections/depmap/"},{"id":"hpa","kind":"collection","name":"Human Protein Atlas","route":"/collections/hpa/"},{"id":"pubmed-europepmc","kind":"collection","name":"PubMed & Europe PMC","route":"/collections/pubmed-europepmc/"}],"idea":[{"id":"idea-tr2-cell-line-passport","kind":"idea","name":"A digital passport for every cell culture: identity, contamination status, passage number","route":"/ideas/idea-tr2-cell-line-passport/"},{"id":"idea-moon-negative-results-ledger","kind":"idea","name":"A global ledger of negative results and failed compounds with mandatory deposition","route":"/ideas/idea-moon-negative-results-ledger/"},{"id":"idea-bio1-negative-preclinical-repository","kind":"idea","name":"A home for the animal and organoid experiments that failed","route":"/ideas/idea-bio1-negative-preclinical-repository/"},{"id":"idea-fund-target-validation-consortium","kind":"idea","name":"A pre-competitive consortium to validate or kill academic targets before licensing","route":"/ideas/idea-fund-target-validation-consortium/"},{"id":"idea-moon-biomarker-validation-utility","kind":"idea","name":"A public biomarker validation utility with pre-diagnostic biobanks and blinded testing","route":"/ideas/idea-moon-biomarker-validation-utility/"},{"id":"idea-tr2-preclinical-negative-registry","kind":"idea","name":"A registry for preclinical experiments that did not work","route":"/ideas/idea-tr2-preclinical-negative-registry/"},{"id":"idea-tr2-ai-external-validation-registry","kind":"idea","name":"A registry of external validation datasets for cancer AI models, with mandatory reporting","route":"/ideas/idea-tr2-ai-external-validation-registry/"},{"id":"idea-tr2-replication-status-badge","kind":"idea","name":"A replication status badge on every cancer paper, visible in PubMed","route":"/ideas/idea-tr2-replication-status-badge/"},{"id":"idea-fund-universal-mta-fast-lane","kind":"idea","name":"A universal material transfer agreement with a thirty-day default","route":"/ideas/idea-fund-universal-mta-fast-lane/"},{"id":"idea-data-accredited-tre-network","kind":"idea","name":"Accredited trusted research environments with curated cancer tables","route":"/ideas/idea-data-accredited-tre-network/"},{"id":"idea-rejuv-core-outcome-set-for-late-effects","kind":"idea","name":"Agree what to measure, so the next systematic review can pool rather than narrate","route":"/ideas/idea-rejuv-core-outcome-set-for-late-effects/"},{"id":"idea-tr2-retraction-propagation","kind":"idea","name":"Alert guidelines and trials when a paper they rely on is retracted","route":"/ideas/idea-tr2-retraction-propagation/"},{"id":"idea-moon-independent-replication-institute","kind":"idea","name":"An independent replication institute that re-tests key preclinical cancer findings before trials","route":"/ideas/idea-moon-independent-replication-institute/"},{"id":"idea-bio1-covalent-ligandability-atlas","kind":"idea","name":"An open map of which cancer proteins any drug can stick to","route":"/ideas/idea-bio1-covalent-ligandability-atlas/"},{"id":"idea-moon-open-cancer-cell-state-model","kind":"idea","name":"An open model of every cancer cell state, built from perturbation atlases","route":"/ideas/idea-moon-open-cancer-cell-state-model/"},{"id":"idea-tr2-arrive-audit","kind":"idea","name":"Audit animal studies for randomisation and blinding, published by institution","route":"/ideas/idea-tr2-arrive-audit/"},{"id":"idea-data-retraction-propagation","kind":"idea","name":"Automatic flagging of retracted or corrected evidence in guidelines and decision support","route":"/ideas/idea-data-retraction-propagation/"},{"id":"idea-tr2-reagent-lot-ledger","kind":"idea","name":"Barcode every reagent lot so batch effects can be traced across experiments and labs","route":"/ideas/idea-tr2-reagent-lot-ledger/"},{"id":"idea-tr2-replication-bounties","kind":"idea","name":"Bounties for documented failed replications of high-impact findings","route":"/ideas/idea-tr2-replication-bounties/"},{"id":"idea-tr2-tenure-replication-credit","kind":"idea","name":"Count replications and open data in hiring and promotion","route":"/ideas/idea-tr2-tenure-replication-credit/"},{"id":"idea-tr2-raw-image-deposit","kind":"idea","name":"Deposit the raw blots, gels and microscopy images behind every figure","route":"/ideas/idea-tr2-raw-image-deposit/"},{"id":"idea-data-ipd-deposit-enforcement","kind":"idea","name":"Enforce individual participant data sharing as a condition of publication and funding","route":"/ideas/idea-data-ipd-deposit-enforcement/"},{"id":"idea-tr2-phd-replication-year","kind":"idea","name":"Every cancer biology PhD begins with a funded replication of a published finding","route":"/ideas/idea-tr2-phd-replication-year/"},{"id":"idea-tr2-reference-compound-panels","kind":"idea","name":"Every drug screen includes standard reference compounds whose performance is published","route":"/ideas/idea-tr2-reference-compound-panels/"},{"id":"idea-tr2-replication-set-aside","kind":"idea","name":"Funders set aside a fixed share of budget for independent replication","route":"/ideas/idea-tr2-replication-set-aside/"},{"id":"idea-bio1-organoid-assay-clinical-validation","kind":"idea","name":"Hold organoid drug tests to the same standard as a diagnostic test","route":"/ideas/idea-bio1-organoid-assay-clinical-validation/"},{"id":"idea-tr2-replication-before-ind","kind":"idea","name":"Independent replication of the key experiment before first-in-human academic trials","route":"/ideas/idea-tr2-replication-before-ind/"},{"id":"idea-tr2-data-link-enforcement","kind":"idea","name":"Journals check that data links actually work, and flag papers whose data vanish","route":"/ideas/idea-tr2-data-link-enforcement/"},{"id":"idea-tr2-preclinical-living-reviews","kind":"idea","name":"Living systematic reviews of animal and organoid evidence before every new trial","route":"/ideas/idea-tr2-preclinical-living-reviews/"},{"id":"idea-bio1-multicentre-mouse-trials","kind":"idea","name":"Multi-centre randomised animal trials before committing to a human trial","route":"/ideas/idea-bio1-multicentre-mouse-trials/"},{"id":"idea-tr2-multilab-preclinical","kind":"idea","name":"Multi-laboratory preclinical trials as the standard for go/no-go decisions","route":"/ideas/idea-tr2-multilab-preclinical/"},{"id":"idea-tr2-antibody-validation-mandate","kind":"idea","name":"Only use antibodies proven to hit their target with knockout controls","route":"/ideas/idea-tr2-antibody-validation-mandate/"},{"id":"idea-tr1-open-protocol-and-sap-library","kind":"idea","name":"Open-source protocol and statistical analysis plan templates with runnable code","route":"/ideas/idea-tr1-open-protocol-and-sap-library/"},{"id":"idea-tr2-paid-statistical-review","kind":"idea","name":"Paid independent statistical review for preclinical papers that inform trials","route":"/ideas/idea-tr2-paid-statistical-review/"},{"id":"idea-bio1-preclinical-preregistration","kind":"idea","name":"Pre-register animal efficacy studies like clinical trials","route":"/ideas/idea-bio1-preclinical-preregistration/"},{"id":"idea-tr2-biomarker-study-registry","kind":"idea","name":"Pre-register biomarker validation studies the way trials are registered","route":"/ideas/idea-tr2-biomarker-study-registry/"},{"id":"idea-data-observational-study-registration","kind":"idea","name":"Pre-registration and results reporting for real-world cancer studies","route":"/ideas/idea-data-observational-study-registration/"},{"id":"idea-tr2-animal-power-mandate","kind":"idea","name":"Pre-specified sample sizes for animal studies; no more 'representative' experiments","route":"/ideas/idea-tr2-animal-power-mandate/"},{"id":"idea-fund-academic-promotion-reform","kind":"idea","name":"Promote academics for trials completed, data shared and findings replicated","route":"/ideas/idea-fund-academic-promotion-reform/"},{"id":"idea-bio1-model-authentication-mandate","kind":"idea","name":"Prove the cell line is what you say it is, or the paper does not run","route":"/ideas/idea-bio1-model-authentication-mandate/"},{"id":"idea-tr2-cell-line-authentication-mandate","kind":"idea","name":"Prove your cell lines are what you say they are, or the paper is not published","route":"/ideas/idea-tr2-cell-line-authentication-mandate/"},{"id":"idea-reg-cart-potency-reference-standards","kind":"idea","name":"Public reference standards and potency assays for CAR-T so every lab measures alike","route":"/ideas/idea-reg-cart-potency-reference-standards/"},{"id":"idea-data-open-weights-for-public-funded-ai","kind":"idea","name":"Publicly funded cancer AI must release open weights and model cards","route":"/ideas/idea-data-open-weights-for-public-funded-ai/"},{"id":"idea-tr2-random-audit","kind":"idea","name":"Random audits of published, funded research, like tax audits","route":"/ideas/idea-tr2-random-audit/"},{"id":"idea-tr2-preclinical-registered-reports","kind":"idea","name":"Registered reports for cancer biology: the plan is peer-reviewed before the result","route":"/ideas/idea-tr2-preclinical-registered-reports/"},{"id":"idea-tr2-automated-stats-check","kind":"idea","name":"Run automated statistics and image checks on every cancer manuscript before review","route":"/ideas/idea-tr2-automated-stats-check/"},{"id":"idea-fund-scoop-protection-policy","kind":"idea","name":"Scoop protection and co-publication norms to reduce academic secrecy","route":"/ideas/idea-fund-scoop-protection-policy/"},{"id":"idea-bio1-model-predictivity-benchmark","kind":"idea","name":"Score every model system on how well it predicted real trial results","route":"/ideas/idea-bio1-model-predictivity-benchmark/"},{"id":"idea-moon-self-driving-cancer-labs","kind":"idea","name":"Self-driving laboratories that run the cancer biology hypothesis loop autonomously","route":"/ideas/idea-moon-self-driving-cancer-labs/"},{"id":"idea-fund-replication-set-aside","kind":"idea","name":"Set aside 3% of grant budgets to replicate findings before translation","route":"/ideas/idea-fund-replication-set-aside/"},{"id":"idea-tr2-reference-model-panels","kind":"idea","name":"Shared reference organoid and PDX panels that every lab can test against","route":"/ideas/idea-tr2-reference-model-panels/"},{"id":"idea-tr2-timestamped-eln","kind":"idea","name":"Time-stamped electronic lab notebooks submitted with the paper","route":"/ideas/idea-tr2-timestamped-eln/"},{"id":"idea-tr2-two-lab-rule","kind":"idea","name":"Top journals require an independent lab to reproduce key findings before publication","route":"/ideas/idea-tr2-two-lab-rule/"}],"section":[{"id":"drug-discovery","kind":"section","name":"Drug Discovery Platforms","route":"/fronts/drug-discovery/"}],"technology":[{"id":"crispr-screens","kind":"technology","name":"CRISPR functional genomics","route":"/technologies/crispr-screens/"},{"id":"functional-drug-testing","kind":"technology","name":"Functional (ex vivo) drug testing","route":"/technologies/functional-drug-testing/"},{"id":"immunonutrition-perioperative","kind":"technology","name":"Immunonutrition before cancer surgery","route":"/technologies/immunonutrition-perioperative/"},{"id":"melatonin-cancer","kind":"technology","name":"Melatonin as a cancer adjunct","route":"/technologies/melatonin-cancer/"},{"id":"organoids","kind":"technology","name":"Patient-derived organoids","route":"/technologies/organoids/"},{"id":"pdx-models","kind":"technology","name":"Patient-derived xenografts","route":"/technologies/pdx-models/"},{"id":"traditional-chinese-herbal-medicine","kind":"technology","name":"Traditional Chinese herbal medicine alongside treatment","route":"/technologies/traditional-chinese-herbal-medicine/"}],"institution":[{"id":"aacr","kind":"institution","name":"American Association for Cancer Research (AACR)","route":"/institutions/aacr/"},{"id":"broad-institute","kind":"institution","name":"Broad Institute of MIT and Harvard","route":"/institutions/broad-institute/"},{"id":"cold-spring-harbor","kind":"institution","name":"Cold Spring Harbor Laboratory","route":"/institutions/cold-spring-harbor/"},{"id":"eacr","kind":"institution","name":"European Association for Cancer Research","route":"/institutions/eacr/"},{"id":"nci","kind":"institution","name":"National Cancer Institute (NIH)","route":"/institutions/nci/"},{"id":"jackson-laboratory","kind":"institution","name":"The Jackson Laboratory Cancer Center","route":"/institutions/jackson-laboratory/"},{"id":"vivli","kind":"institution","name":"Vivli","route":"/institutions/vivli/"},{"id":"yoda-project","kind":"institution","name":"Yale University Open Data Access (YODA) Project","route":"/institutions/yoda-project/"}],"paper":[{"id":"paper-begley-nature","kind":"paper","name":"Drug development: Raise standards for preclinical cancer research","route":"/key-papers/paper-begley-nature/"},{"id":"paper-reproducibility-project-cancer-biology-elife-2021","kind":"paper","name":"Reproducibility Project: Cancer Biology found that landmark preclinical results mostly shrank or vanished on replication","route":"/key-papers/paper-reproducibility-project-cancer-biology-elife-2021/"},{"id":"paper-freedman-plos-biol","kind":"paper","name":"The Economics of Reproducibility in Preclinical Research","route":"/key-papers/paper-freedman-plos-biol/"}],"roadmap":[{"id":"ai-oncology-roadmap","kind":"roadmap","name":"AI in oncology roadmap: pattern readers → foundation models → agents in the workflow","route":"/roadmaps/ai-oncology-roadmap/"},{"id":"drug-discovery-roadmap","kind":"roadmap","name":"Drug discovery roadmap: screening in mice → maps of dependency → designing in silico","route":"/roadmaps/drug-discovery-roadmap/"}]}}