I HAVE a confession to make: some of the articles that have appeared in New Scientist, including ones I have written, are wrong. Not because we deliberately misled you. No, our reports were based on research by respected scientists at top universities, published in peer-reviewed journals. Yet, despite meeting all the normal standards of credibility, some findings turned out to be false.
Science is in the throes of what is sometimes called the replication crisis, so named because a big hint that a scientific study is wrong is when other teams try to repeat it and get a different result. While some fields, such as psychology, initially seemed more liable than others to generate such “fake news”, almost every area of science has since come under suspicion. An entire field of genetics has even turned out to be nothing but a mirage. Of course, we should expect testing to overturn some findings. The replication crisis, though, stems from wholesale flaws baked into the systems and institutions that support scientific research, which not only permit bad scientific practices, but actually encourage them. And, if anything, things have been getting worse over the past few decades.
Advertisement
Yet as awareness of the problem has grown, so have efforts to tackle it. So, how are these opposing forces faring? Will the efforts to combat fake science succeed? And how can you know if the research you read about in New Scientist and elsewhere will ever make it out of the lab and start working in the real world?
It is hard to pinpoint when the replication crisis began, but many people got their first inkling of it in 2011. That year, three things happened to ring alarm bells. First, a study was published that claimed to demonstrate psychic abilities. Some people apparently experienced improvements to their memory of word lists if they were given reminders after being tested. In other words, . Later that year, a paper showed . The key is “cherry-picking”, producing lots of data and only using the figures that confirm your hypothesis, a practice also sometimes called p-hacking, after one of the terms in a commonly used statistical technique. A third blow was the brewing scandal over Dutch psychologist Diederik Stapel, who was caught simply making up results. This showed that the much-vaunted journal peer-review system – where journals ask experts in a field to decide whether a paper should be accepted – is no guarantee that only good science gets published.
“That was the year that psychology had this wake-up call,” says psychologist . What’s more, there were growing rumblings that some seminal psychology results were unreliable. In 2012, researchers that people exposed to stereotypes of ageing will walk more slowly when leaving the lab. The original experiment was a key result in the field of “priming”, the idea that people’s behaviour is affected by unconscious cues. Failure to replicate was also the undoing of another textbook finding, that we can , so if we are trying to resist tempting food, we find it harder to persevere with a mental task. Both these once-influential ideas have now fallen into disrepute.
For some science journalists like myself, these U-turns and the ensuing rows were fascinating, but not that concerning. Psychology has long been seen as less rigorous than the “harder” life sciences, like medicine or cell biology, mainly because human behaviour is so complex, making it difficult to measure and theorise about. But I was being too complacent. In the past decade, unreplicable scientific claims have emerged elsewhere – even in the physical sciences. It pains me to say it, but my own beat of medicine and the other life sciences seem among the most affected.
One of the worst hit areas is early stage biomedical research: laboratory studies of potential new drugs tested on animals or in cells grown in a dish. Such research is regularly taken up by pharmaceutical firms, who have the resources to turn it into new medicines. But it often turns out to be unsound. In 2011, an internal survey at drugs firm Bayer of the leads from university research were failing to stand up when tested by company scientists. The next year, biotech firm Amgen investigated 53 landmark papers and found that only .
This, for me, was a real shocker. Many people understand that promising results in mice may not translate to people. Before this, I didn’t realise that results in mice may not even translate to other mice. Of course, not all university biomedical research is wrong – it has been the basis for many life-enhancing inventions, not least vaccines against covid-19. But I hate to think how many stories I have written that gave people false hope. Cancer research seems particularly hard hit, with a project published in 2021 confirming how few papers in this area can be replicated. All that waste of researchers’ time, all those animals that died for nothing, not to mention the money, often raised by the public through fun runs and bake sales.
If you think the solution is to pay attention only to research in people, rather than in cells or animals, forget it. Several fields of medical research carried out in people also have reputations for unreliable findings. Take nutrition research, which aims to find out which foods are healthiest. It has become a standing joke that, depending on the day of the week, you can find studies telling you that a given food is bad for you or good. When I wrote an article about this in 2019, saying we should be sceptical of all nutrition studies, some colleagues thought I must be overstating the problems. If anything, my mistake was to confine my criticisms just to food research.
This state of affairs shouldn’t really be surprising because the same causes of bad science in psychology are turning up in many places. Cherry-picking is increasingly possible because of the vast amounts of data generated in research these days, as a result of greater computing power to store and analyse it.
Noisy numbers
The more complex the science, the more data you get – like in brain scanning. Despite producing compelling images showing different parts of the brain “lighting up” in response to different stimuli, what actually comes out of an fMRI scanner is a mass of numbers. Researchers must then work out changes in brain activity from changes in blood flow, while factoring in multiple other variables. “We have to remove all of the noise in the scanner, noise in the way that our body’s responding… and tens of other complications,” says at the University of Oxford.
Initial brain scanning studies used too few human subjects to ensure that real patterns in the data stood out from the noise. Matters came to a head in 2009 with a study apparently showing that pictures of humans in emotional situations could trigger neural activity in a dead Atlantic salmon. The researchers had used the fish to test that their scanner worked, but then wrote up the . “That was an important message,” says Gould van Praag.
But MRI researchers still have problems. A bombshell study found last month that research on links between the brain’s structure or function and complex mental traits – such as cognitive abilities or risk of depression – would need thousands of participants. The average number of people scanned is 23.
Another way that the scientific record gets distorted is publication bias: studies that find a positive result are more likely to be published than those that don’t. It isn’t just scientists who tend to brush negative results under the carpet – so do editors and peer reviewers of journals. A less-talked-about kind of publication bias is the preference for headline-generating results. If you think journal editors should be above caring whether their papers are exciting, think again. The publishing industry has a ranking system called impact factors to indicate a journal’s kudos. Impact factors are based on how many times each journal’s papers have been cited in other papers. So “groundbreaking” research that gets more citations is highly prized.
The lure of fame has probably tempted some into bad science, but over recent years, the pressures have grown on scientists to produce as many papers in high-ranking journals as they can. One contributor to this ethos, sometimes known as publish or perish, is the increasingly competitive nature of academia, with growing numbers of junior researchers competing for limited jobs.
Its foundation is the way academic success is measured. Universities and the bodies that fund research – usually governments or medical charities – want to assess scientists’ performance. That is hard, so they ask scientists how many papers they have published and what the journal impact factors were. “We are incentivised to publish and get grants,” says of Bristol, UK, who heads the , a group of researchers who are trying to improve things. “Those are the things that get us jobs, get us promoted. It doesn’t matter if the findings are wrong.”
These problems collectively led to the rise and fall of an entire field of biology called candidate gene association studies. From the late 1990s, there were headline-grabbing claims that specific versions of certain genes active in the brain cause various mental health conditions. The field spawned hundreds of media headlines announcing that scientists had found the gene “for” depression, schizophrenia, aggression and so on. But then technology improved and researchers started doing more laborious studies involving hundreds of thousands of people, analysing not just individual genes but all the participants’ DNA. These showed that common conditions like depression are affected by hundreds of gene variants, each with a tiny effect. None of the results from candidate gene studies stood up. “It was a lot of bunkum,” says , Ireland.
Cycle of dysfunction
The more I researched this article, the more I came to despair. The problems are driven by three sectors that seem locked in a self-perpetuating cycle of dysfunction. Journals want flashy results to boost their impact factors and gain prestige. Funders award grants to people who have published in the most prestigious journals. And universities hire those who bring in the big grant money.
And yet there are efforts under way to break the cycle. They aren’t being led, as you might expect, by the ancient and venerable institutions of science. In fact, change has been driven mainly by junior researchers trying to fix things from the bottom up. “These have been people working in their bedrooms, creating websites, running Twitter campaigns,” says Orben, who does such work herself.
One approach is “preregistering”, where researchers describe their hypothesis and their intended study on a website before they begin. It means a negative result can’t just be ignored. With enough detail about the methods, it can also prevent cherry-picking. Some journals now let papers display a symbol to show that the study was preregistered. The , a US grassroots organisation that tackles bad science, has created . It also offers two more badges indicating that researchers will share all their data and information on methods and materials, allowing others to check or replicate the study.
In a more radical step, a preregistered study can be sent to a journal before it is carried out, a system called registered reports. If peer reviewers OK it, the research gets published no matter what the results, further reducing the risk that negative studies get airbrushed from history. “The journal accepting the paper on the basis of its method rather than its results is transformative,” says psychologist Stuart Ritchie at King’s College London. “This is the best way to do science.”
Currently, only a small fraction of papers are preregistered or submitted as registered reports. Perhaps unsurprisingly, progress seems fastest in the field of psychology. Nevertheless, I am excited about these practices, known collectively as open science. If they catch on more widely, it would be a simple way for anyone to judge which claims are most reliable.
Slowly, creakingly, there are signs that funding bodies are trying to address the problem too. The National Institutes of ҹ1000, the biggest funder of biomedical research in the US, says that from next year all grant recipients . UK Research and Innovation (UKRI), the umbrella body for the seven research councils, which fund most science in UK universities, has similar ambitions, although no start date yet. Speaking in February at an inquiry into reproducibility held by the House of Commons science and technology committee, UKRI chief executive – and she is hugely influential.
Meanwhile, an international agreement called the asks universities and funders to pledge not to use journal impact factors in hiring, promotion or grant decisions. Many heads of universities have signed up, but it remains to be seen whether this will influence middle managers who make hiring decisions, says Orben. Another signatory is , the world’s largest charitable funder of cancer research. So maybe those fun runs are worth it after all.
I asked everyone I interviewed for this article how much progress has been made in fixing bad science since those alarm bells started ringing more than a decade ago. Nearly all say that a good start has been made – but there is a long way to go. Munafò puts it slightly differently. “There’s a saying that revolutions happen very slowly and then all at once,” he says. “It feels like we’re getting to the all-at-once stage where real change is actually happening.” Let’s hope he is right.
