An Empirical Study to Understand How Students Use ChatGPT for Writing Essays

Abstract: As LLMs advance and become widespread, students increasingly turn to systems like ChatGPT for assistance with writing tasks. Educators are concerned with students' usage of ChatGPT beyond cheating; using ChatGPT may reduce their critical engagement with writing, hindering students' learning processes. The negative or positive impact of using LLM-powered tools for writing will depend on how students use them; however, how students use ChatGPT remains largely unknown, resulting in a limited understanding of its impact on learning. To better understand how students use these tools, we conducted an online study $(n=70)$ where students were given an essay-writing task using a custom platform we developed to capture the queries they made to ChatGPT. To characterize their ChatGPT usage, we categorized each of the queries students made to ChatGPT. We then analyzed the relationship between ChatGPT usage and a variety of other metrics, including students' self-perception, attitudes towards AI, and the resulting essay itself. We found that factors such as gender, race, and perceived self-efficacy can help predict different AI usage patterns. Additionally, we found that different usage patterns were associated with varying levels of enjoyment and perceived ownership over the essay. The results of this study contribute to discussions about how writing education should incorporate generative AI-powered tools in the classroom.

• The paper identifies distinct usage patterns—planning, translating, reviewing, and full delegation—as central to ChatGPT-assisted essay writing.
• The study uses an online experiment with 70 students to reveal demographic and self-efficacy predictors influencing ChatGPT usage for writing tasks.
• Findings suggest ChatGPT increases essay length while mixed usage promotes higher ownership and satisfaction in the writing process.

An Empirical Study on Student Utilization of ChatGPT in Essay Writing

The paper "An Empirical Study to Understand How Students Use ChatGPT for Writing Essays" explores the increasingly prevalent use of LLMs like ChatGPT in student writing. With LLMs impacting various domains, including education, the study seeks to empirically understand how students use these tools in writing assignments. This research provides an essential examination of LLM usage and reflects the attitudes toward and implications of such technologies in an educational setting.

The authors conducted an online study involving 70 college students who completed an essay-writing task using a customized platform designed to capture their interactions with ChatGPT. By tracking queries sent to ChatGPT and subsequent writing behaviors, the study aims to answer four primary research questions: the nature of student usage of ChatGPT in writing, predictive factors for ChatGPT usage patterns, how usage manifests in essay writing, and perceptions of the writing experience facilitated by ChatGPT.

Key Findings

1. Patterns and Categories of Usage: The study identified four main categories of student interactions with ChatGPT, informed by Flower and Hayes' cognitive writing model: Planning, Translating, Reviewing, and delegating full Writing tasks (All). The study's results indicate significant variance in ChatGPT usage across students, suggesting diverse motivations and strategies for leveraging AI in essay writing.
2. Demographic Predictors: The research highlighted that individual differences, such as gender, race, and self-efficacy in writing, could predict how frequently students engage ChatGPT. For example, students with lower self-efficacy in writing were more likely to rely on ChatGPT across all categories except for full writing tasks. Contrarily, there were no significant predictors found for the All category, indicating that other factors might be at play when students choose to delegate their writing entirely to ChatGPT.
3. Impact on Writing Output: The use of ChatGPT was associated with increased word counts in essays, particularly for students who used it for full writing tasks. However, the authors reported no significant difference in user-contributed text across user types. These findings suggest that while ChatGPT aids in generating more content, the interactive depth of student involvement in writing varies by individual usage strategy.
4. Perception and Experience: Students' perceptions of the essays they wrote with ChatGPT's assistance differed across user types. The study found that those who used ChatGPT primarily for planning reported higher ownership of their written work compared to those who let ChatGPT write full parts of their essays. Notably, a mixed usage approach fostered greater enjoyment in the writing process, emphasizing the importance of user engagement in improving the subjective writing experience.

Practical and Theoretical Implications

The empirical insights from this study have significant implications for pedagogical strategies and the integration of AI tools in writing education:

• Integration in Pedagogy: The research suggests the potential for integrating LLMs like ChatGPT in educational settings as assistants for planning, ideation, and review, rather than solely for content generation. Educators could leverage these tools to enhance learning objectives by encouraging critical engagement with writing tasks.
• Policy and Guidance Development: Findings reinforce the need for developing policies and guidelines regarding AI usage in academia. A deeper understanding of AI interaction patterns can assist educators in framing nuanced rules on ethical AI usage while fostering exploratory learning environments.
• Broader AI Adoption Discussions: By highlighting demographic predictors and variances in LLM usage, the study enriches discussions on AI acceptance and its diverse impact on educational methodologies. Future research can extend beyond student perceptions and assess AI's broader impact on academic integrity and skill development.

Future Research Directions

The study opens avenues for future work by raising questions about how ChatGPT's integration into academia can balance skill development with technological reliance. It suggests investigating strategies that foster authentic learning experiences and academic honesty amidst widespread AI adoption. Additionally, exploring gender and cultural nuances in AI interactions can yield insights into optimizing AI tools for inclusive educational practices.

In summary, this study offers a comprehensive empirical investigation into how students use ChatGPT for essay writing. Its findings elucidate the complex interplay between LLMs and writing processes, providing a foundation for discussions on the evolving educational landscape influenced by advanced AI technologies.