Of Data, Devils, and Details

A recent preprint study has plenty of gremlins in its data, perhaps enough to invalidate the conclusions

Of Data, Devils, and Details

Are you ready for some post-publication review?

A couple of weeks ago, I wrote about two studies touting preprints as reliable, both of which seemed a little bizarre while exhibiting some publication problems of their own.

The first paper touting preprints as reliable, published in BMJ Medicine, didn’t include the article’s customary Review History. The editorial team is still trying to retrieve this, weeks later. Then, when I requested data from the authors via email, they claimed to have relied solely on a data source not listed in the paper.

The editor has been informed of the discrepancy in methods reporting.

The second paper touting preprints as reliable, in The Lancet: Global Health, didn’t include the list of articles covered, except in a table without enough detail to allow any secondary analyses. I requested the data from the corresponding author, got a bit of a run-around about social justice-like issues, finally complained to the editor, and last week received from the corresponding author a list of PMIDs for the articles included in the study.

Using the PMIDs to look at various publication events for these preprint-paper pairs, I found some data discrepancies, and a fundamental flaw — one which I consistently find with such analyses (see Twitter feed, below, for evidence that this is a consistent and predictable flaw).