The Problem of Preprints in the Press

Preprint servers can initiate the spread of un-peer-reviewed information in the media, which can influence public opinions and affect public health.

Last month, I learned that a New York Times story about voters with a “need for chaos” was based on a psyRxiv preprint that was a year old and had yet to be published in a peer-reviewed journal. Here was non-peer-reviewed information being promulgated in an esteemed newspaper as if it were a solid, peer-reviewed study, and it was quoted back to me by a high-achieving, socially powerful person who had internalized the information.

This made me wonder how often the press is using un-peer-reviewed papers from preprint servers, and bioRxiv in particular — which often hosts preprints dealing with human health — as a basis or source for news stories. Could such stories mislead the public? Could preprints fool journalists?

Using a tool called “AHRefs,” I analyzed the backlinks to bioRxiv from four major news outlets: the New York Times, the Washington Post, the LA Times, and the Guardian. The tool isn’t perfect, but it gave me enough interesting data to talk about here.

Across these sources, I found 43 total news stories linking to bioRxiv preprints. Of these, 20 stories were based on or cited preprints that to-date have no associated peer-reviewed publication. Some of these preprints date back to 2015. That leaves 23 news stories linking to bioRxiv that cited preprints that resulted in peer-reviewed publications.

Based on this sample, 46% of bioRxiv preprints cited in these four news outlets represent scientific claims that have not been peer-reviewed.

The stories based on these include one that created quite a stir — a story in 2016 which started another “cell phones cause cancer” scare, a story which the American Cancer Society still has to publicly rebut.

You may remember this story, which caused many people to believe their brains were being cooked by their cell phones. As Aaron E. Carol memorably wrote in the New York Times debunking the preprint (and documenting how many major news sources covered the preprint):

Recently, I lost a day at work when my . . . colleague Austin Frakt emailed me first thing in the morning to tell me that headlines were appearing declaring that an “explosive new cellphone-cancer” study was making the rounds. As I have long been interested in this topic, I started to read the headlines and news.

“Cellphone-Cancer Link Seen in Rat Study,” said Time. “Cellphone Radiation Linked to Cancer in Major Rat Study,” said IEEE Spectrum, a magazine for engineers. I was dismayed to say the least. “Game-changing,” as a quotation in a Mother Jones headline put it, seemed like a bit of an overreach.

So I went ahead and read the paper. Despite what some outlets reported [Wall Street Journal], this was a not-yet-published study of rats that had been shopped for “review,” but had not been accepted by any editors.

In this short space, we learn that a preprint fooled reporters for Time, IEEE Spectrum, Mother Jones, and the Wall Street Journal. In no case was the study mentioned as a preliminary study or preprint. It was instead called “a massive federal study” or “[a]n important new study.” IEEE Spectrum’s portrayal of this un-reviewed science was especially breathless over the preprint, with its subhead reading:

The “gold-standard” research found a low incidence in brain and heart tumors, but only in male rats. Expect regulatory agencies to react, says expert.

Too bad this “gold standard” research was just a preprint — one that has yet to generate a peer-reviewed paper in a professional journal, and one that yielded findings that have been largely rejected as flawed and irrelevant.

A more recent public health-related issue involving bioRxiv has to do with the “sonic attack” against US embassy personnel in Cuba, which a preprint claimed resembled lovelorn crickets. While the preprint was published nearly 10 months ago, it has yet to result in a peer-reviewed publication. Does this un-reviewed research create a false sense of security, trivializing what appears to have been a sonic attack on embassies? Wouldn’t something with worldwide import — remember, a similar attack was reported in China later — have benefited from waiting to see if other scientists and experienced editors thought it was potentially valid?

Even our friend, the “Neuroskeptic,” has stumbled into the preprint brambles, publishing a story in Discover magazine about an un-peer-reviewed preprint in 2017. The preprint has not yet resulted in a paper in a peer-reviewed journal, a fact that does not augur well for its quality or future. Neither does its premise, which claimed to identify:

. . . the existence of “functional connectivity” between surgically disconnected distant brain regions using fMRI, something that in theory shouldn’t be possible.

In other words, the preprint hypothesized magic. And, because of bioRxiv, it has received not only unfettered online distribution and discovery, but led an experienced science journalist down the garden path.

In what appears to be a contradiction, our friends at “RetractionWatch” regularly amplify preprints, some of which have ended up published in peer-reviewed journals, some of which have not — even years after posting. What they pick to amplify comports with their view of the world, from what I can tell, suggesting that preprints can feed “filter bubbles” and confirmation biases. The first one I cited above — about “chaos” thinkers in politics — likely feeds confirmation biases among New York Times’ readers. But to see critics of bad information themselves falling prey to information that in some cases is never published in an editorially-reviewed and peer-reviewed journal — while holding these journals to standards of accountability — seems a bit of a double-standard.

Not that journals are perfect. Peer-reviewed and editorially-reviewed papers have occasionally misled the public as well, usually resulting in scandals, retractions, and so forth. This level of accountability seems to be missing from preprints. Why haven’t preprints that have misled the public been retracted? Why are there no COPE guidelines for handling controversies around preprints? Why is there no accountability for these purveyors of publicly available information? Where is “RetractionWatch” in cases like this?

Coverage of non-peer-reviewed science in these outlets has a few consequences. First, other media pick up these stories. For instance, the story about crickets in Cuba was picked up at Berkeley because it represented, as the writer notes:

. . . a conclusion quickly picked up by the New York Times and zipped around the internet.

These news outlets — especially the New York Times and the Washington Post — provide news coverage into dozens or hundreds of downstream papers. While I didn’t test how extensively this occurred, the amplification potential is significant. These news outlets are also highly ranked in search results.

It’s also worth noting that even when there was a peer-reviewed publication available for linking, reporters often linked to preprints, suggesting there is confusion about what a preprint represents.

This all undermines the value and place of peer-review, which is supposed to be a filter between claims and published articles. By blurring the line about what is “published,” preprint servers devalue editorial review and peer-review, even if they say they believe otherwise. When a draft paper receives nearly all the distribution and branding benefits of a peer-reviewed paper, including media coverage, the hurdle of peer-review is undercut as something that must be cleared to realize the benefits of publication.

As noted above, some bioRxiv preprints have to be debunked by journalists. That’s good journalism, but it shows there is burden and risk created by making draft papers available publicly.

The potential for preprints to add confusion seems to be something the managers of bioRxiv’s even riskier spinoff, medRxiv, seem to appreciate. They have placed the following on their site in prominent red type:

Caution: Preprints are preliminary reports of work that have not been peer-reviewed. They should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.

Given the way some preprints are being amplified in the popular press — and non-peer-reviewed science is being presented as confirmed and finalized, leading to at least one memorable public health scare — it may be time for bioRxiv to add a similar warning to its site. It may also be time for COPE and others to create processes to make preprint services accountable for what they publish.

Subscribe now