Does Formatting "Cost" $1,908?
A paper suggests formatting is expensive for academics, but it's a very weak study
This weekend, there was a study from PLOS One cited on Twitter. The authors attempted to assess how much it “cost” to format papers for submission. A quote from the conclusion was trumpeted via tweet, and retweeted with abandon. But in actually reading the study, I came away convinced this is a very shaky study.
Now, I’m all for eliminating “formatting” requirements for authors when it comes to styling references and dealing with margins and line spacing. But I don’t think “formatting” a table accurately, or an illustration or chart precisely, or ensuring references are accurately placed and don’t wander, or a set of image panels so they correctly match their captions, will ever be a waste of time. What “formatting” is in this context isn’t clear, it’s probably not usually a waste of time, and that’s a major problem for the authors’ premise.
The authors indicate that the study included 372 participants from 41 countries. But when you get into the data, you see that 223 of these were from one country — Canada — which leaves 149 participants from 40 countries. But 80% of the respondents came from 5 countries — Canada, Australia, Spain, the US, and the UK. Only 70 participants came from outside these 5 countries. On top of this, the sample skews female far more than the general population of scientists — estimates put science gender distribution at 70/30 male/female, while the sample in this paper was 44/56 male/female, which suggests the sample isn’t that representative. Also, nearly 18% of the sample were students, not graduates or working scientists or scholars, and the authors didn’t compensate for that somewhat inexplicable and potentially irrelevant subset. The authors used “snowball sampling,” which introduces biases as it depends on chaining from the authors to colleagues to other colleagues. Finally, the survey was open for 4 months, which flies in the face of the authors’ claim that the survey was up “for a short period of time,” and suggests the snowball probably melted somewhere along the way, leaving pretty watery data.
As noted above, a main conceptual problem is with the definition of “formatting” used in the survey instrument. There is no definition. It could mean anything — data formatting, formatting supplemental appendices, incorporating information from co-authors and adjusting formats for accuracy afterwards.
A main interpretation problem is this:
Our results suggest that each manuscript costs 14 hours, or US $477 to format for publication in a peer-reviewed journal. This represents a loss of 52 hours or a cost of US$1,908 per person-year
The authors act as if “formatting” has no intellectual or work value, and is always a waste of time. The $1,908 is portrayed as wasted money.
Because the definition of “formatting” isn’t clear, and given the hours and the possible tasks that could be interpreted as “formatting” — writing, revising, preparing data tables, developing figure panels — “formatting” probably means far more than just superficial work.
As author groups have become more collaborative, that means more cats to herd for corresponding authors. Is that part of the time spent? Does “formatting” involve incorporating comments from 3-7 other authors? Does it involve fixing a table that blew up because Journal 1 found a data problem? Doesn’t Journal 2 deserve a well-formatted table? The authors make no attempt to probe how much time in the bin “formatting” has to do with collaborative work, data formatting, textual formatting, reference formatting, writing, or revisions. As a result, they impose their hypothesis on the data, instead of gathering the data needed to prove or disprove their hypothesis.
Let’s assume that there is some intellectual work involved in the activities the survey respondents are describing. Aren’t scientists and clinicians paid to do intellectual work? Isn’t this sometimes also physical work (e.g., stirring mixtures, weighing compounds, running centrifuges, assembling machinery)? Don’t scientists think while they manipulate materials, run experiments, and handle mice and other critters? Aren’t those things tedious and repetitive, too? As a result, do they all waste money?
Is formatting/writing a paper, formatting/revising a paper, and formatting/revising an accepted manuscript really “a loss” of hours? Is formatting a table or a dataset a waste of time? Or is it just more of the mix of physical and intellectual work scientists do every day?
After reading the paper, seeing the survey instrument, thinking about the results, and working through the interpretation, I think we have here a weak study from a potentially non-representative sample — 18% students, gender mix off, mostly one country, definitely not geographically diverse.
There are other issues with the paper. The title — “Scientific sinkhole: The pernicious price of formatting” — rings of amateurism and sensationalism to me. The authors aren’t even trying to be objective. In addition to the kind of headline a teenager might write, the authors are allowed to boast in the paper, claiming multiple times to have published the “first” paper estimating such costs. They are also allowed to manipulate perceptions, with sentences like, “we were able to recruit a large and diverse sample, suggesting that this is an area of importance and interest within the scientific community” — as noted above, the sample wasn’t very large, was mostly Canadian, was 80% from 5 countries, didn’t match community gender demographics, and had a large proportion of students in it.
Does formatting “cost” $1,908 per person per year? Who knows? These authors don’t, and this paper is yet another bit of shaky information being amplified on Twitter. Get ready for the anti-format movement. There’s no vaccine for it, unfortunately.