Friday, March 14, 2014
On the problem of irreproducible data
Given the potential that stem cells hold for curing a number of human diseases, it's no surprise that two high-profile papers published in the January edition of Nature garnered a great deal of attention. The papers basically suggested that creating stem cells from normal ones could be as simple as exposing them to acid or mechanical stress, such as forcing cells through glass tubes. Even more, these STAP (stimulus-triggered acquistion of pluripotency) cells seemed to be some sort of "super stem cell", capable of forming not only any body tissue but also the placenta, something that other pluripotent cells normally cannot do.
Most stem cell biologists were taken aback at the simplicity of the method. George Daley, a leading stem cell biologist at Harvard, called the findings "fascinating", "perplexing" and "begging to be replicated". Another researcher, Dieter Egli, bluntly said that if he described the method to his colleagues, they would think he was kidding.
Turns out they called it right. Several researchers now report that they are unable to replicate the original findings. Qi-Long Ying (whose lab was one building over from mine at USC), an original supporter of the method, was one of the unsuccessful ones. There's also a blog keeping record of other failed attempts. When the original author, Haruko Obokata (RIKEN/Harvard) and her colleagues shared the methodology they used, it turns out the process wasn't as simple as it initially sounded, and in fact was technically quite challenging. Subsequently, some of the data in the papers has also come under question, and calls for retraction have already been issued. (Edited to add: Uh oh.)
This isn't the first high-profile set of experiments that have failed the independent replication test, nor will they be the last. But this ties in very neatly to one of the things that frustrated me most during my Ph.D. - being able to reproduce the same set of data multiple times, until you're absolutely sure that your findings are real. Some of my work was performed in primary endothelial cells, and as any life scientist can tell you, working with primary cells is never easy. They can be terribly finicky creatures, and two different batches of cells may give completely different results in the same experiment. The number of repeats you have to run to get statistically significant data keeps increasing, and when you're repeating an experiment for the nth time, it can get really annoying.
Or when you're a grad student, trying to repeat a supposedly very easy protocol published in a journal like Nature (so it must be true!) and you can't get it to work, it's easy to ask yourself whether you're the one with the problem. Maybe you're not doing it right, or you just have bad hands, or you need to stand on one leg and recite the Lord's prayer backwards when performing the fifth step in the protocol. Who knows? Enough to drive even the most even-keeled of people slightly mad.
This is an issue that I am sure a lot of other life scientists struggle with, and something that I plan to revisit over the course of a few blog posts. In the meantime, I leave you with a fabulous compendium of articles on this issue published as a special issue by Nature a few years ago. Excellent, and very thought-provoking indeed. As we perform increasingly technically sophisticated experiments using better and bigger equipment, issues of reliability and reproducibility become all the more important. If we want to uphold the high ethical standards that all research is supposed to meet, these are things that we need to think, talk and debate about just as keenly as we discuss which lab got published on Nature's cover page last week.