AI Review Paper Is Plain Quackers

A quick post to get it off my chest.

I was forwarded a working paper by two members of Harvard’s Institute for Quantitative Social Science with the file name “peerless.pdf.”

The lead author is Gary King. The second is Steve Worthington.

Their paper is misguided in many ways:

1. They assert peer-review is a quality assurance (QA) step in editorial work. It is not. It is first and foremost a selection step, with editors first judging whether a submission is relevant and worth spending time on (authors are qualified, work seems to be up to current standards, authors tackle an interesting topic).
   1. From this point onward, peer review is advisory to the editorial process.
   2. “Peer” means another person with relevant expertise and experience capable of advising on the merits and relevance of a study, it’s design and assumptions, and it’s overall interest level, with some possible expert insights into problems or limitations with the reporting, data, elements (missing or underdeveloped), or conclusions
2. They rely on a data set gleaned from an internal review system at Harvard where reviewers are explicitly told not to evaluate relevance, appropriateness, interest, or whether the research meets a standard. They are asked to just provide helpful feedback to aid a revision.
   1. This is worse than operationalizing their mistaken view of peer review. It turns the reviewers into toothless sycophants and makes any study of the data irrelevant to the day-to-day editorial work journals do under resource constraints.
      1. Worse, they thank “various AI models” for helpful comments. 🤢🤮
3. Their main goal — reading between the lines as a human understanding incentives and sneaky academic games and tricksters — seems to be to create an environment where desk rejects happen less often and more papers get published with AI accelerating this and reducing initial decision barriers by experts who might find the work lacking in more important ways.

The working paper is fundamentally flawed by these conceptual problems and obvious conflicts of interest (despite protestations), things that a human peer-reviewer would have caught in an instant — I did, the person sending it to me did, and so forth. This paper as it stands should be rejected, and requires a major revision before resubmission.

And then things got funny.

King’s web site has a chatbot (GaryAI 🤢🤮). I asked it if this working paper was his, and here’s the exchange:

Yet, when you go to his “Writings” listing on the same site, here it is:

And AI is supposed to be more accurate as a QA element in peer review . . .

So, the bottom line is:

1. The framing of the paper is wrong, the data is gathered from a source based on this same flawed premise, and the overall work is misleading because the people who wrote it lack qualifications and knowledge while also wanting to push aside desk reject decisions to get more of their AI slop through
2. The AI on one of the author’s own sites can’t accurately respond to a simple question about his writings, even though the information is clearly available

Please stop touting AI in peer review. AI is not a peer. Review isn’t QA. You Mssrs. Evans and Worthington do not know what you’re talking about.