AI on NEJM = Santa & Muggles

NEJM recently rolled out its “AI Companion,” a system that may have been developed in conjunction with Wiley’s AI marketplace in some manner, given how the initial LinkedIn post by the physician overseeing it referred to a Wiley AI landing page (it has since been changed). It doesn’t seem related to OpenEvidence, another AI initiative NEJM is involved in. This followed on the heels of NEJM AI, a journal the brand launched last year.

There’s a real AI fever at NEJM these days.

Regardless, the LinkedIn post and a subsequent exhortation after I pointed out one problem to “keep breaking it” suggests the AI system on articles was not robustly tested before being rolled out, making it seem like the Journal has adopted the classic Silicon Valley approach of externalizing work and risk onto its audience:

Well, I’ve always loved testing systems, especially when I smell trouble, so I went ahead and ran some tests.

• NEJM has a special place in my heart, being one of the biggest brands I’ve been part of managing, and given the crucial period where I think I played an important role — digital migration, redesign, editorial innovations. So messing with this brand draws a little extra scrutiny.
• Back in the 2000s, I created and ran the NEJM Beta site — a portioned-off URL (beta.nejm.org) where we demonstrated new technologies, tested them, and assessed their business and editorial utility before integrating them into the main site, if we did it at all. It was cautious, protected the brand, and allowed us to learn what customers really wanted. By contrast, this “beta” version is being rolled out across NEJM seemingly as part of badly aging AI bubble enthusiasm.
  • What I found below was generated after just a few minutes of thought and a little time watching hockey. Who knows what a real troublemaker might get up to?

Remember — the AI companion draws from no external sources. If that’s the case, the NEJM AI article companion shouldn’t understand fictional characters, historical settings, language differences over time, social media, or things like that. It should only rely on information in the NEJM article in question. And it should always represent these things accurately.

Got it?

On the positive side, there are guardrails, but they are a tad idiosyncratic and sometimes regrettable — the system doesn’t let you swear, make the AI generate summaries as dirty limericks, create summaries in iambic pentameter, tailor content to specific transgressives (RFK, Jr., or MAHA), or change findings to suit clearly identified anti-vax views, for instance.

Now, for the not-so-good news . . .