Perceptions and Detection of AI Use in Manuscript Preparation for Academic Journals

Abstract: The emergent abilities of LLMs, which power tools like ChatGPT and Bard, have produced both excitement and worry about how AI will impact academic writing. In response to rising concerns about AI use, authors of academic publications may decide to voluntarily disclose any AI tools they use to revise their manuscripts, and journals and conferences could begin mandating disclosure and/or turn to using detection services, as many teachers have done with student writing in class settings. Given these looming possibilities, we investigate whether academics view it as necessary to report AI use in manuscript preparation and how detectors react to the use of AI in academic writing.

• The paper reveals that only 22% support reporting AI grammar corrections while 52% endorse disclosure when manuscripts are rewritten using AI.
• The paper finds that surveys show minimal differences between ChatGPT and research assistant support, yet notable distinctions with paid tools like Grammarly and Word.
• The paper demonstrates that AI detectors often flag grammar fixes as AI-generated, indicating challenges in accurately distinguishing editing types.

The paper "Perceptions and Detection of AI Use in Manuscript Preparation for Academic Journals" (2311.14720) investigates academics' perceptions of the necessity of reporting AI use in manuscript preparation and how AI detectors react to AI in academic writing.

Perceptions of Academics on Reporting AI Use:

• Academics are less likely to believe that using AI to fix grammar in manuscripts should be reported compared to using AI to rewrite manuscripts. The survey found that 22% of respondents thought grammar correction should be reported, while 52% felt text rewriting should be.
• There was little difference in reporting preferences between using ChatGPT and research assistant (RA) help for both grammar correction and text rewriting.
• Significant differences in reporting preferences were observed between ChatGPT/RA assistance and paid proofreading, Grammarly, and Word. Academics were less inclined to believe that the use of Grammarly, Word, or proofreading services should be reported.
• Disagreements exist among academics regarding whether using ChatGPT to rewrite text needs to be reported, and these differences are related to perceptions of ethics, academic role (professor vs. student/postdoc), and English language background. Native English speakers and professors were less likely to believe that using ChatGPT to rewrite text should be reported.

AI Detector Reactions:

• The AI detection software (Originality.ai) studied did not always distinguish between AI-assisted grammar correction and text rewriting. Abstracts revised using GPT-3.5 to fix grammar were often flagged as having a high likelihood of being AI-written.
• The distribution of AI scores was skewed more towards higher scores for abstracts rewritten using the "Rewrite 1" prompt compared to those revised with the "Grammar 1" prompt, although both produced high AI scores.
• The AI detector was able to distinguish between manuscripts that were not revised by ChatGPT and those that were revised.