Sunday, April 22, 2012

Irreproducibility in cancer research

Over the past decade, before pursuing a particular line of research, scientists... in the haematology and oncology department at the biotechnology firm Amgen in Thousand Oaks, California, tried to confirm published findings related to that work. Fifty-three papers were deemed 'landmark' studies. It was acknowledged from the outset that some of the data might not hold up, because papers were deliberately selected that described something completely new, such as fresh approaches to targeting cancers or alternative clinical uses for existing therapeutics. Nevertheless, scientific findings were confirmed in only 6 (11%) cases. Even knowing the limitations of preclinical research, this was a shocking result. ...

Unfortunately, Amgen's findings are consistent with those of others in industry. A team at Bayer HealthCare in Germany last year reported that only about 25% of published preclinical studies could be validated to the point at which projects could continue. Notably, published cancer research represented 70% of the studies analysed in that report, some of which might overlap with the 53 papers examined at Amgen. ...

Some non-reproducible preclinical papers had spawned an entire field, with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis. More troubling, some of the research has triggered a series of clinical studies — suggesting that many patients had subjected themselves to a trial of a regimen or agent that probably wouldn't work.

Table 1: Reproducibility of research findings
Preclinical research generates many secondary publications, even when results cannot be reproduced.
Journal impact factorNumber of articlesMean number of citations of non-reproduced articles*Mean number of citations of reproduced articles
Results from ten-year retrospective analysis of experiments performed prospectively. The term 'non-reproduced' was assigned on the basis of findings not being sufficiently robust to drive a drug-development programme.
*Source of citations: Google Scholar, May 2011.
>2021248 (range 3–800)231 (range 82–519)
5–1932169 (range 6–1,909)13 (range 3–24)

--C. Glenn Begley (former global head of Hematology and Oncology Research at Amgen) and Lee M. Ellis, Nature, on one reason the war on cancer isn't progressing quickly