Cadu Rolim/Fotoarena/ZUMA
Over the past few months, there has been a pretty heated debate raging in both scientific and political circles over the antimalarial drug hydroxychloroquine. The broad outlines are likely familiar at this point: President Trump has touted its effectiveness to treat COVID-19, even claiming to be taking it himself as a preventative measure, without scientific evidence to back him up and even though the FDA has warned against its use outside of hospital settings.
Now there’s a new development: On Thursday, researchers retracted a major study that suggested a link between hydroxychloroquine and a substantially increased risk of death. Another leading journal has issued a statement that questions the integrity of related research.
The now-retracted study, which published last month in the Lancet, a respected research journal, included analysis of 96,032 patients’ records and was at the time the largest study to investigate the drug’s potential in fighting COVID-19. According to the study, patients who received a cocktail of hydroxychloroquine and an antibiotic saw a 45 percent increased risk of death. The news was a big deal. Several media organizations, including Mother Jones, picked up the study. The World Health Organization, which had been investigating hydroxychloroquine’s potential as a coronavirus treatment, paused recruitment for its trial. And Italy, France, and Belgium banned the drug from being used to treat COVID-19.
Still, questions about the study started popping up shortly after it published; more than 180 researchers issued a letter raising potential problems with the data behind it, and on May 29, the Lancet corrected some of the data. Now, just shortly after the Lancet and the New England Journal of Medicine published “expressions of concern” about the research, the authors have officially retracted the paper.
“Today, three of the authors of the paper, ‘Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis’, have retracted their study,” the Lancet announced in a statement. “They were unable to complete an independent audit of the data underpinning their analysis. As a result, they have concluded that they ‘can no longer vouch for the veracity of the primary data sources.'”
NEJM, which published a study on May 1 drawing from the same data source, wrote in an expression of concern, “Recently, substantive concerns have been raised about the quality of the information in that database. We have asked the authors to provide evidence that the data are reliable.”
Without access to the data, it’s unclear what exactly, if anything, went wrong, though I recommend you read this excellent investigation by the Guardian into the company that provided the dataset. But what we do know now is that the controversy is concerning—if not unsurprising, given how fast the science is moving around COVID-19 and how desperate people are for information on the disease that’s claimed more than 100,000 American lives.
As I’ve written before, experts say the pandemic has put a strain on an already flawed scientific review process. The coronavirus has brought on a deluge of manuscripts for journals to review. Multiple journals, including NEJM, told me in April that their peer-review process has been expedited. But even in normal times, peer-reviewed journals often don’t review raw data as part of the vetting process in the first place. “I would argue maybe they should more often because that might prevent cases like this,” Ivan Oransky, a founder of Retraction Watch, a blog that tracks academic retractions, told me earlier this week after the Lancet study had been flagged for concern, but before it was retracted; Oransky also serves as the vice president of editorial at medical news and reference site Medscape and teaches medical journalism at New York University. “Journals hardly ever look at the underlying data.” Other times bias and error simply slip by peer-reviewers.
When Mother Jones asked the Lancet via email on Thursday how this paper made it past peer-review, the journal did not answer directly. It instead underscored that the paper was, in fact, peer-reviewed. “All research articles published in The Lancet journals undergo independent, external peer review, including statistical review,” it wrote. (NEJM simply told Mother Jones that the expression of concern “must speak for itself for now.”)
The Lancet study in question is one of a handful of COVID-related papers that have been retracted already, though this might be the most high-profile. So far, Retraction Watch has documented 14 retractions (including today’s) and two temporary retractions for recent coronavirus-related research, as well as two expressions of concern about COVID-19 papers (including NEJM‘s notice this week).
Generally speaking, expressions of concern are not common in the science world, but they’re not unheard of either. “An expression of concern is used when editors have a reasonable sense that there might be something wrong but they’re not sure yet. So it’s not a retraction, but it’s grander than a correction,” says Oransky. Retractions are more serious. On average, according to Oransky, the amount of time between when a paper is published and withdrawn is about three years. Again, the pandemic has put this entire timeline into super-speed.
Both papers using this now-disputed data were authored by a team of researchers led by Dr. Mandeep Mehra, a professor of medicine and a cardiovascular specialist at Harvard Medical School. A co-author is Sapan Desai, who leads Surgisphere, the data firm which supplied the COVID dataset for the studies. The Guardian investigation raises a slew of concerns about the validity of the data, including significant data discrepancies and conversations with hospitals that said they did not participate. It also reports that Desai “has been named in three medical malpractice suits, unrelated to the Surgisphere database” and that several of the company’s employees have “little or no data or scientific background.” (Again, it’s worth reading the Guardian‘s investigation in full.) Despite pleas from the scientific community, the authors of the initial study have yet to supply the raw data from the studies, citing confidentiality concerns.
Complicating matters on hydroxychloroquine even further, the first major controlled clinical trial of the drug was published on Wednesday in the New England Journal of Medicine and found that it does not help prevent infection. As the New York Times reports:
The malaria drug hydroxychloroquine did not prevent Covid-19 in a rigorous study of 821 people who had been exposed to patients infected with the virus, researchers from the University of Minnesota and Canada are reporting on Wednesday…. Conducted in the United States and Canada, this trial was also the first to test whether the drug could prevent illness in people who have been exposed to the coronavirus.
…
“The take-home message for the general public is that if you’re exposed to someone with Covid-19, hydroxychloroquine is not an effective post-exposure preventive therapy,” the lead author of the study, Dr. David R. Boulware, from the University of Minnesota, said in an interview.
Though both dispute the effectiveness of hydroxychloroquine, this new NEJM study and the now-retracted Lancet study have highly different methodologies. The Lancet paper was observational: The researchers had no control over who received or did not receive treatment. A controlled clinical trial, on the other hand, which happened in the new NEJM study, is a much more rigorous experiment (though it’s still not foolproof).
If there’s one definitive takeaway this week it’s that the research is moving quickly. And a lot of it is incomplete. At a time when we need good science the most, experts tell me bad science is slipping through the cracks at an alarming rate. “What [the pandemic] has done is just made everyone rush to publication and rush to judgment, frankly,” Oransky told me in April. “You’re seeing papers published in the world’s leading medical journals that probably shouldn’t have even been accepted in the world’s worst medical journals.”
