The trouble with rushing ‘science’

A major scandal shook the medical community recently, with the New England Journal of Medicine (NEJM) and the Lancet — two of the most prestigious medical journals — retracting highly-influential articles on COVID-19 because the global database they were based on turned out to be dubious.

In the first place, scholars wondered how Surgisphere, a company with no track record and just a few staff, could get hold of a dataset involving almost 100,000 patients in six continents. As it turned out, many of the supposed partner institutions had never heard of the company. The accuracy of the data was flagged, too; for instance, it included cases from Africa even before the first COVID-19 case was reported.

The Lancet paper, which found that hydroxychloroquine was associated with a higher risk of death and complications among coronavirus patients, was a particularly consequential publication, given that it was used as the basis by WHO and other institutions to suspend clinical studies on a drug touted by world leaders like Jair Bolsonaro and Donald Trump as a treatment for the coronavirus.

Eventually it emerged that various other studies were based on the same suspicious database, whose owner (and coauthor in the studies), Dr. Sapan Desai, had an equally suspicious record of making various claims (e.g. designing a “wearable neural induction device”). One study, on Ivermectin, had not yet been published but was already available as a preprint. That too, was retracted, and as I write this, more studies are being investigated.

The Surgisphere scandal raises serious questions about what exactly constitutes “science” and how it is used. Needless to say, we need all the information we can get about an unknown virus, and with such an aim in mind, journals have scrambled to churn out publications.

But is the mad dash to publish works contributing to a diminished criticality — and, without meaning to, lowered standards?

On top of peer-reviewed works, we are also deluded with “preprints” that, despite not being reviewed, are already circulating, like the above-mentioned Ivermectin study. One preprint, for instance, claimed that BCG vaccine is associated with low incidences and deaths from COVID-19, a provocative claim that — although unsubstantiated and unvested — has circulated around the world. The practice of preprints has been around for a long time, but never before under the intense gaze of journalists waiting to write the next big headline.

As the Surgisphere papers show, these academic works have real-life effects, and so do their retractions, especially in a time of pandemic. A study about the nature of COVID-19 transmission, for instance, can alter the quarantine and distancing measures of entire nations, and determine the viability of entire practices (e.g. dining in restaurants, conducting worship services).

On the other hand, a retraction such as the ones involving Surgisphere can erode trust in science as a whole, ultimately undermining all kinds of research no matter how rigorously done.

Thankfully, journals are already pledging to learn from the episode. “In the future, our review process of big data research will include reviewers with such specific expertise,” pledged the editors of the NEJM. Concerns, however, are not limited to big data research. The peer review process itself needs soul-searching, and so is the mediation of scientific knowledge by news outlets and politicians who are quick to report studies to sell headlines and boost political capital — at the expense of accuracy.

This is particularly true for news related to cures or vaccines. The steroid dexamethasone, for instance, was quickly hailed as a “life-saving treatment,” creating the impression that it was a COVID-19 cure — even though all it did was to lower mortality among those already critical ill.

As for vaccines, an effective one can save the world from the pandemic and definitively establish trust for the idea of immunization, but if a study showing the efficacy of vaccines turns out to be problematic, it will do far more harm by stoking vaccine hesitancy. (Ironically, the contemporary anti-vaccine movement was encouraged in large part by a 1998 Lancet paper, likewise since retracted, that erroneously associated measles vaccine and autism.)

If we are to be guided by “science,” then the institutions that produce, publish, and act on it must make sure that it is truly science in substance, not just in form.

[email protected]