Loading
An eight-year effort to repeat 50 experiments from 23 high-impact cancer biology papers found that replication effect sizes were on average 85% smaller than the originals, and fewer than half of the effects replicated by most criteria.
The Center for Open Science and Science Exchange set out in 2013 to replicate selected experiments from 53 high-impact cancer biology papers published 2010-2012, with registered reports and original-author consultation. Only 50 experiments from 23 papers could be completed; the companion paper (Errington et al., eLife 2021, 'Challenges for assessing replicability') documents that no original paper contained enough methodological detail to design a replication without contacting the authors, and that about a third of authors were unhelpful or unresponsive.
Across 158 measured effects, the median replication effect size was 85% smaller than the original; 92% of replication effects were smaller than the originals. Using five criteria, 46% of effects replicated on more criteria than they failed; for original positive results, about 40% replicated, while null results replicated at 80%.
The project quantified the preclinical reproducibility problem that pharmaceutical groups (Begley and Ellis 2012; Prinz 2011) had reported anecdotally, and shaped funder requirements for rigour and data sharing.
Many exciting laboratory findings that motivate drug programmes are weaker or less reliable than published, which helps explain the high failure rate of drugs entering clinical trials. It argues for pre-registration, detailed methods, data sharing and independent replication before major translational investment.
Shares Patient-derived xenografts, Preclinical results do not reproduce, Patient-derived organoids, The valley of death between lab and product.
Shares Patient-derived xenografts, Preclinical results do not reproduce, Lab models that fail to predict what happens in patients, The valley of death between lab and product.
Shares Patient-derived xenografts, Preclinical results do not reproduce, Patient-derived organoids, Lab models that fail to predict what happens in patients.
Shares Preclinical results do not reproduce, Patient-derived organoids, Lab models that fail to predict what happens in patients, The valley of death between lab and product.
Shares Patient-derived xenografts, Preclinical results do not reproduce, Patient-derived organoids, Lab models that fail to predict what happens in patients.
Shares Patient-derived xenografts, Preclinical results do not reproduce, Lab models that fail to predict what happens in patients.
Shares Preclinical results do not reproduce, Lab models that fail to predict what happens in patients, Failures are hidden.
Shares Patient-derived xenografts, Patient-derived organoids, Lab models that fail to predict what happens in patients, Failures are hidden.