Monya Baker

More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments. Those are some of the telling figures that emerged from Nature’s survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research.
The data reveal sometimes-contradictory attitudes towards reproducibility. Although 52% of those surveyed agree that there is a significant ‘crisis’ of reproducibility, less than 31% think that failure to reproduce published results means that the result is probably wrong, and most say that they still trust the published literature.

3 thoughts on “Monya Baker

  1. shinichi Post author

    1,500 scientists lift the lid on reproducibility

    Survey sheds light on the ‘crisis’ rocking research.

    by Monya Baker

    Nature 533, 452–454

    (26 May 2016)

    http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970

    More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments. Those are some of the telling figures that emerged from Nature’s survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research.

    The data reveal sometimes-contradictory attitudes towards reproducibility. Although 52% of those surveyed agree that there is a significant ‘crisis’ of reproducibility, less than 31% think that failure to reproduce published results means that the result is probably wrong, and most say that they still trust the published literature.

    Data on how much of the scientific literature is reproducible are rare and generally bleak. The best-known analyses, from psychology1 and cancer biology2, found rates of around 40% and 10%, respectively. Our survey respondents were more optimistic: 73% said that they think that at least half of the papers in their field can be trusted, with physicists and chemists generally showing the most confidence.

    The results capture a confusing snapshot of attitudes around these issues, says Arturo Casadevall, a microbiologist at the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland. “At the current time there is no consensus on what reproducibility is or should be.” But just recognizing that is a step forward, he says. “The next step may be identifying what is the problem and to get a consensus.”

    Failing to reproduce results is a rite of passage, says Marcus Munafo, a biological psychologist at the University of Bristol, UK, who has a long-standing interest in scientific reproducibility. When he was a student, he says, “I tried to replicate what looked simple from the literature, and wasn’t able to. Then I had a crisis of confidence, and then I learned that my experience wasn’t uncommon.”

    The challenge is not to eliminate problems with reproducibility in published work. Being at the cutting edge of science means that sometimes results will not be robust, says Munafo. “We want to be discovering new things but not generating too many false leads.”

    The scale of reproducibility

    But sorting discoveries from false leads can be discomfiting. Although the vast majority of researchers in our survey had failed to reproduce an experiment, less than 20% of respondents said that they had ever been contacted by another researcher unable to reproduce their work. Our results are strikingly similar to another online survey of nearly 900 members of the American Society for Cell Biology (see go.nature.com/kbzs2b). That may be because such conversations are difficult. If experimenters reach out to the original researchers for help, they risk appearing incompetent or accusatory, or revealing too much about their own projects.

    A minority of respondents reported ever having tried to publish a replication study. When work does not reproduce, researchers often assume there is a perfectly valid (and probably boring) reason. What’s more, incentives to publish positive replications are low and journals can be reluctant to publish negative findings. In fact, several respondents who had published a failed replication said that editors and reviewers demanded that they play down comparisons with the original study.

    Nevertheless, 24% said that they had been able to publish a successful replication and 13% had published a failed replication. Acceptance was more common than persistent rejection: only 12% reported being unable to publish successful attempts to reproduce others’ work; 10% reported being unable to publish unsuccessful attempts.

    Survey respondent Abraham Al-Ahmad at the Texas Tech University Health Sciences Center in Amarillo expected a “cold and dry rejection” when he submitted a manuscript explaining why a stem-cell technique had stopped working in his hands. He was pleasantly surprised when the paper was accepted3. The reason, he thinks, is because it offered a workaround for the problem.

    Others place the ability to publish replication attempts down to a combination of luck, persistence and editors’ inclinations. Survey respondent Michael Adams, a drug-development consultant, says that work showing severe flaws in an animal model of diabetes has been rejected six times, in part because it does not reveal a new drug target. By contrast, he says, work refuting the efficacy of a compound to treat Chagas disease was quickly accepted4.

    The corrective measures

    One-third of respondents said that their labs had taken concrete steps to improve reproducibility within the past five years. Rates ranged from a high of 41% in medicine to a low of 24% in physics and engineering. Free-text responses suggested that redoing the work or asking someone else within a lab to repeat the work is the most common practice. Also common are efforts to beef up the documentation and standardization of experimental methods.

    Any of these can be a major undertaking. A biochemistry graduate student in the United Kingdom, who asked not to be named, says that efforts to reproduce work for her lab’s projects doubles the time and materials used — in addition to the time taken to troubleshoot when some things invariably don’t work. Although replication does boost confidence in results, she says, the costs mean that she performs checks only for innovative projects or unexpected results.

    Consolidating methods is a project unto itself, says Laura Shankman, a postdoc studying smooth muscle cells at the University of Virginia, Charlottesville. After several postdocs and graduate students left her lab within a short time, remaining members had trouble getting consistent results in their experiments. The lab decided to take some time off from new questions to repeat published work, and this revealed that lab protocols had gradually diverged. She thinks that the lab saved money overall by getting synchronized instead of troubleshooting failed experiments piecemeal, but that it was a long-term investment.

    Irakli Loladze, a mathematical biologist at Bryan College of Health Sciences in Lincoln, Nebraska, estimates that efforts to ensure reproducibility can increase the time spent on a project by 30%, even for his theoretical work. He checks that all steps from raw data to the final figure can be retraced. But those tasks quickly become just part of the job. “Reproducibility is like brushing your teeth,” he says. “It is good for you, but it takes time and effort. Once you learn it, it becomes a habit.”

    One of the best-publicized approaches to boosting reproducibility is pre-registration, where scientists submit hypotheses and plans for data analysis to a third party before performing experiments, to prevent cherry-picking statistically significant results later. Fewer than a dozen people mentioned this strategy. One who did was Hanne Watkins, a graduate student studying moral decision-making at the University of Melbourne in Australia. Going back to her original questions after collecting data, she says, kept her from going down a rabbit hole. And the process, although time consuming, was no more arduous than getting ethical approval or formatting survey questions. “If it’s built in right from the start,” she says, “it’s just part of the routine of doing a study.”

    The cause

    The survey asked scientists what led to problems in reproducibility. More than 60% of respondents said that each of two factors — pressure to publish and selective reporting — always or often contributed. More than half pointed to insufficient replication in the lab, poor oversight or low statistical power. A smaller proportion pointed to obstacles such as variability in reagents or the use of specialized techniques that are difficult to repeat.

    But all these factors are exacerbated by common forces, says Judith Kimble, a developmental biologist at the University of Wisconsin–Madison: competition for grants and positions, and a growing burden of bureaucracy that takes away from time spent doing and designing research. “Everyone is stretched thinner these days,” she says. And the cost extends beyond any particular research project. If graduate students train in labs where senior members have little time for their juniors, they may go on to establish their own labs without having a model of how training and mentoring should work. “They will go off and make it worse,” Kimble says.

    What can be done?

    Respondents were asked to rate 11 different approaches to improving reproducibility in science, and all got ringing endorsements. Nearly 90% — more than 1,000 people — ticked “More robust experimental design” “better statistics” and “better mentorship”. Those ranked higher than the option of providing incentives (such as funding or credit towards tenure) for reproducibility-enhancing practices. But even the lowest-ranked item — journal checklists — won a whopping 69% endorsement.

    The survey — which was e-mailed to Nature readers and advertised on affiliated websites and social-media outlets as being ‘about reproducibility’ — probably selected for respondents who are more receptive to and aware of concerns about reproducibility. Nevertheless, the results suggest that journals, funders and research institutions that advance policies to address the issue would probably find cooperation, says John Ioannidis, who studies scientific robustness at Stanford University in California. “People would probably welcome such initiatives.” About 80% of respondents thought that funders and publishers should do more to improve reproducibility.

    “It’s healthy that people are aware of the issues and open to a range of straightforward ways to improve them,” says Munafo. And given that these ideas are being widely discussed, even in mainstream media, tackling the initiative now may be crucial. “If we don’t act on this, then the moment will pass, and people will get tired of being told that they need to do something.”

    Reply
  2. shinichi Post author

    科学の「再現性」が危機に瀕している

    by Carlos Henrique

    http://gigazine.net/news/20160531-scientists-reproducibility/

    Natureの調べによると、調査対象となった1576人研究者のうち、自分以外の科学者が発表した実験を再現するのに失敗したことがある研究者の割合は70%以上で、全体の50%以上は自分自身が行った実験を再現するのに失敗しているとのこと。

    各分野ごとの再現失敗経験は以下のような感じ。化学、生物学、物理学&エンジニアリング、薬学、地球環境学、その他、という分野ごとに調査結果をわけたところ、化学と生物学における再現失敗の割合が比較的多くなっていることがわかります。

    そして、「科学の再現性は危機に瀕しているか?」という質問に「はい、かなり危機に瀕している」と答えたのは52%で、「はい、少し危機に瀕している」という回答は38%、「危機はない」という回答は7%となっており、多くの研究者が危機を感じていました。一方で、「再現できなかった論文は内容が間違っている」と考えている科学者は31%以下で、多くは発表された論文が信頼に値すると考えているようです。これは、多くの科学者が「実験が再現できない」ということを過去に経験しているからであると考えられます。

    また、今回の調査に回答した研究者のうち73%は、自分たちの研究分野において発表され科学雑誌に掲載されたた論文のうち、少なくとも50%は信頼に値すると考えているとのこと。発表された研究内容に最も信頼を置いているのは化学分野と物理学&エンジニアリングの分野でした。

    ただし、現代の科学における「再現性」がどうあるべきか、という定義ははっきりしていません。科学の正確性を保つためにも、「何が問題で再現がどうあるべきかという合意がなされる時が来ているのかもしれない」とJohns Hopkins Bloomberg School of Public Healthの微生物学者であるアーチュロー・カサデバル教授は語っています。

    また、「他の科学者の実験を再現できなかった」という事実は自分の無能さの現れとして受け取られかねないことや、自分の研究内容を漏らしてしまう可能性があることなどから、「実験が再現できなかった」と本家の研究グループに伝えたことがある人は全体の20%にとどまっており、これも問題の1つとして挙げられます。

    そして、既に行われた研究を再現するための研究を発表しようとする人がそもそも少なく、その上、「再現できない」という結果に終わった場合、ネガティブな内容を避けたがる出版社が論文を掲載したがらない、ということも科学の再現性における障害となっています。

    再現研究を科学誌に掲載しようとした研究者たちの、掲載確率は以下のような感じ。再現に成功した研究のうち、科学誌に掲載された割合は24%で、掲載を断られた割合は12%となっています。一方で、再現に失敗した研究の場合は掲載される確率が低く、13%にとどまりました。

    回答者のうち、「過去5年間において研究所で実験の再現を推進するための具体的な取り組みが行われた」と回答したのは全体の3分の1で、分野ごとに見てみると、取り組みに対して最も消極的だったのは物理学&エンジニアリングの分野で24%、積極的だったのは薬学の分野で41%でした。取り組みの具体的な内容として挙げられたもののうち、最も多かった事例は「もう一度自分で実験を行う」というものと、「研究所内の誰かに再現を頼む」というものとなりました。

    研究の再現において何が障害となるのか、という質問項目に対しては、60%以上の回答者が「選択的な報告」と「出版に対するプレッシャー」との2つを「常に/しばしば障害になっている」ものとして選択。また、研究所における検証の不足や、監督不足、検出力の低さなどを「常に/しばしば障害になっている」として挙げたのは全体の半数以上で、再現における技術的な難しさが問題点だと考えている人は少数だったとのこと。

    では、再現のプロセスを改善するには何が必要か?というアンケートに対して、最も選ばれた回答は「統計的データをもっと理解する」「指導・監督を改善する」「実験の計画をより強固にする」といったもので、回答者全体の90%以上がこれらの案に賛同。奨励金を設けたり、出版社側のチェックの目をよりよくするという案は再現のプロセスを改良するアイデアとしては、人気の低いものとなりました。

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *