Navigating AI in Peer Review: Balancing Potential with Ethical Safeguards

Ask any peer reviewer or editor what's changed most about their workload in the last two years, and generative artificial intelligence (AI) is likely to come up quickly in conversation. Peer reviewers might identify AI tools as something they've started using for extra review help, or they’re thinking about whether to do this and how to do it ethically. Editors will say they’re seeing more and more formulaic-appearing peer reviews that look suspiciously like they are AI-generated. Authors also are noticing this in peer reviews too. 

One survey of researchers suggests that over half of reviewers have already incorporated AI tools somewhere in their review workflow. This gap between informal use and disclosed, sanctioned practice raises a critical question for this year’s Peer Review Week theme, "Peer Review Capacity: Volume, Speed, and Quality": How can the scholarly publishing ecosystem and wider scientific community expand capacity and speed without sacrificing quality?

JMIR Publications' position starts from a simple premise: AI use could be a legitimate part of a reviewer's toolkit, but only as long as peer reviewers carefully consider key guiding principles before they accept an invitation to review a manuscript. 

First, maintaining confidentiality of unpublished work is non-negotiable and reviewers are explicitly agreeing to this scientific norm and editorial policy when accepting an invitation to peer review. Uploading an unpublished manuscript, in whole or in part, into a commercial AI tool – even just to get a faster summary or a second opinion on the writing – can mean sending confidential material to a third-party system whose terms of service say nothing about protecting it. JMIR Publications' peer review AI use guidance permits limited AI use under specific safeguards grounded on ensuring confidentiality.

Second, the human reviewer is accountable as the scientific expert who is using an AI tool to help them; essential rules of thumb here include validating AI outputs and not trying to pass off such material as originally and solely created by the human reviewer. While JMIR Publications has not banned AI use outright, responsible AI use still should not substitute for a reviewer's own expert judgment. Editors and authors want to know what human expert reviewers think – not what AI says.

Finally, transparency – meaning full AI use disclosure – ensures proper attribution and explanation of the peer review report’s origins. JMIR Publications’ generative AI use best practices treats accountability, transparency, and confidentiality as the same trio of guidance that applies to reviewers just as much as authors. A reviewer using AI to help structure a report, check for consistency, or flag potential issues should say so, and should remain the party accountable for the substance of the review. AI can assist a reviewer's thinking; it should not stand in for it.

AI’s Potential in Peer Review

Realizing AI's potential requires addressing essential ethical considerations to protect the scientific record and maintain trust. When used responsibly, AI tools offer tangible opportunities to strengthen peer review capacity by augmenting human capability:

• Increased efficiency: AI can assist in structuring reports, verifying completeness, or checking language and formatting, freeing experts to focus on complex methodological and substantive evaluations.

• Enhanced consistency: Automated checks can help reviewers triage obvious issues and ensure key checklist items or reporting guidelines are systematically addressed across submissions.

AI tools used well (e.g., for organizing thoughts, checking completeness, or triaging obvious issues) can genuinely help a reviewer and ensure timeliness in completing a good review. AI tools used carelessly, without confidentiality safeguards, accountability, or disclosure, risk eroding trust, creating inconsistent or poor-quality reviews, creating new work for journal editors trying to catch what went wrong, and ultimately becoming a disservice to hard-working researchers submitting their work for review and publication.

Addressing volume, speed, and quality simultaneously requires viewing each of these three elements as interconnected towards the same end goal. Thoughtfully applied AI can help reviewers manage rising submission volumes efficiently, but speed must never come at the expense of confidentiality, disclosure, rigorous standards, and human accountability.

Read JMIR's 2023 editorial Best Practices for Using AI Tools as an Author, Peer Reviewer, or Editor and JMIR's policy on generative AI tool use JMIR Publications Editorial Policy on the use of generative AI during manuscript preparation.

Useful Tips

• Never upload a manuscript, in whole or in part, to a commercial AI tool. Doing so risks exposing confidential, unpublished work to a system with no obligation to protect it.

• AI can help but it cannot replace your expert judgment. You remain the named, accountable party for your review report. If AI contributed to how it was drafted or organized, you are still responsible for its accuracy and fairness.

• Disclose AI use in your review process, just as you would disclose a conflict of interest. Transparency about your process protects your credibility as much as it protects the author's work.

• Used thoughtfully, AI can help manage rising review volume without cutting corners, but accountability, confidentiality, and transparency (disclosure) are not optional trade-offs for speed.

For more information: 

Navigating the Intersection of AI and Peer Review: A Guide for Ethical Integration

The Art of Peer Review: Ensuring Quality and Validity in Research

Inclusive Language in Peer Reviews: Fostering Respect and Equity in Academia

Ethical Considerations in eHealth & Informatics Research: The Reviewer's Role