From the window, I still note the presence of winter. I grab my blanket to avoid the cold draft’s frigid grasp. Low and behold, I start seeing small pops of color piercing the stark white blanket of snow. Crocuses of every shade imaginable herald the long-awaiting news of the winter’s ending reign of tyranny. The crocuses also foretell the ominous arrival of … bunnies!
Yes, bunnies. As adorable as they are, they can be a gardener’s greatest foe, decimating a lovely spring garden. Every year, the fluffy army grows larger and larger. How is it that these critters can reproduce with such great ease in exponential quantities?
Not everything reproduces as easily as bunnies. It may seem hard to believe, but scientific experiments can be difficult to replicate. But why? The reasons are as bountiful as the numbers in the fluffy horde waging war on spring flowers.
When trying to reproduce another lab’s research, the actual published paper can present hurdles. The often-overlooked methods section may omit key details crucial for the success of the project. These “tips and tricks” might include simple techniques, such as warming a solution to get a powder to readily dissolve. Other times, the source and catalog number of the reagents get omitted. This information might be key if different vendors have different manufacturing strategies or if slightly similar products are available. In addition, authors write the methods section sparsely to meet the character/word limit set by a journal. A call to action has been raised for standardizing the writing of the methods section (Erdemir, 2013).
Sometimes, the reproducibility problem arises from the reagents themselves unbeknownst to the authors/researchers. Not too long ago, researchers learned that mice and rats presented different behaviors during an experiment depending on the gender of the person carrying out the experiment (Sorge et al., 2014). In another example, antibodies can be a source for data variability and headaches (see an earlier blog for expanded discussion on the matter).
To make things even more complex, the weather or location of the lab can affect the experimental outcome. In the crystallography field, stories exist where crystal quality could not be reproduced, with suspicions that the elevation difference or humidity level was responsible. Weather pitfalls are not limited to crystallography, but ordinary bench work as well. Problems can arise if the humidity is too high and causes the unexpectant breakdown of a chemical reagent.
So, how big of a problem is reproducibility in science? A pretty big one, actually. Amgen tried to reproduce the findings from fifty-three published papers, but only succeeded 11 % of the time (Begley & Ellis, 2012). Bayer HealthCare also encountered a similar problem and could only replicate 25% of the findings from their selected studies (Begley & Ellis, 2012). Instances such as these have motivated the creation of the Reproducibility Project: Cancer Biology. So far, the collaboration only could reproduce the findings from 2 of the 5 selected papers (Kaiser, 2017).
If findings cannot be replicated, how useful are they? What should the science community do?
The solutions may be as bountiful as the members of the fluffy brigade. We could follow the suggestion of the call to action for standardizing the methods section of papers. Also, we could upload well written lab notebooks (that include the “tips and tricks,” maybe atmospheric conditions and details about the experimenter for animal studies) to a website for all to see. Maybe the best thing to do, however, would be to reduce the emphasis of being the first to publish a novel finding. Instead, we should reward heavily the research groups that take the time to replicate previous papers, trouble shoot the reasons why a finding could not be reproduced, and publish their findings.
Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483(7391), 531-533. doi:10.1038/483531a
Erdemir, F. (2013). How to write a materials and methods section of a scientific article? Turk J Urol, 39(Suppl 1), 10-15. doi:10.5152/tud.2013.047
Kaiser, J. (2017). Rigorous replication effort succeeds for just two of five cancer papers. Retrieved from http://www.sciencemag.org/news/2017/01/rigorous-replication-effort-succeeds-just-two-five-cancer-papers
Sorge, R. E., Martin, L. J., Isbester, K. A., Sotocinal, S. G., Rosen, S., Tuttle, A. H., . . . Mogil, J. S. (2014). Olfactory exposure to males, including men, causes stress and related analgesia in rodents. Nat Methods, 11(6), 629-632. doi:10.1038/nmeth.2935