Evaluating Large Language Model Creativity from a Literary Perspective

Abstract: This paper assesses the potential for LLMs to serve as assistive tools in the creative writing process, by means of a single, in-depth case study. In the course of the study, we develop interactive and multi-voice prompting strategies that interleave background descriptions (scene setting, plot elements), instructions that guide composition, samples of text in the target style, and critical discussion of the given samples. We qualitatively evaluate the results from a literary critical perspective, as well as from the standpoint of computational creativity (a sub-field of artificial intelligence). Our findings lend support to the view that the sophistication of the results that can be achieved with an LLM mirrors the sophistication of the prompting.

• The paper demonstrates that careful prompting enables LLMs to generate increasingly sophisticated creative text.
• The study employs dialogue, temperature adjustments, and multi-voice generation to critically evaluate narrative creativity.
• The findings suggest that interactive AI tools can significantly enhance creative writing while challenging traditional literary norms.

Insights into Evaluating LLM Creativity from a Literary Perspective

This paper, titled "Evaluating LLM Creativity from a Literary Perspective," conducts a case study to explore the potential of LLMs as tools for assisting in creative writing, evaluating them from both literary and computational creativity angles. The study highlights three main approaches: creative dialogue, varying sampling temperature, and a multi-voice generation experiment. The findings suggest that the sophistication of the output from an LLM closely correlates with the complexity and creativity embedded in the user prompts.

Summary of the Investigations

The focal point of the research is a speculative fiction project, involving a 16th-century character, Effie, who experiences a time-travel-induced encounter with the 21st century. The authors used this narrative as a testbed to examine various interactions with a LLM (GPT-4), seeking to derive insights into the model's creative capacities.

Techniques Explored

1. Creative Dialogue: Here, the LLM interacts in a dialogue format where the human plays the critic or mentor, creating an iterative process of drafting and refinement. The study emphasizes the potential of LLMs to improve textual drafts through structured feedback and stylistic suggestions. Analysis revealed the ability of the model to produce increasingly sophisticated text as interactions progressed.
2. Raising the Temperature: This technique involves altering the temperature parameter of the model to change the randomness of the text output. Lower temperatures result in more deterministic responses, while higher temperatures foster creative, albeit sometimes risky and unintelligible outputs. This parameter tweaking unleashes the model’s capability to generate text that ventures into unpredictable, experimental linguistic territory, useful for sparking creative ideas.
3. Multi-Voice Generation: The authors created a dialogue within the model itself by simulating both an "author" and a "mentor" role. This approach demonstrated the model's capacity to provide self-assessment and critique, showcasing its potential for autonomous role-playing and multi-character narrative construction, which introduced a new character, Margaret, into the narrative effortlessly.

Critical Evaluation

The study critically evaluates the model outputs using traditional literary criticism methods, focusing on style, imagery, narrative consistency, and thematic elements. The research underscores the importance of interaction between human and AI, positing that LLMs can significantly contribute to creative writing when effectively guided by sophisticated, contextual prompts.

The paper argues that this interactive potential exhibits a form of computational creativity, where the model reveals a degree of autonomy and introduces novel elements into its narratives, pushing the usual boundaries of human–AI collaboration in literature.

Implications and Future Outlook

The implications of this research extend into multiple domains. Practically, this paper suggests that LLMs could be incorporated as interactive tools in creative industries to supplement and enhance human creativity. Theoretically, it raises questions about the role of computational entities in narrative creation, blurring the lines between human-authored and machine-authored texts.

Looking forward, such studies encourage further development of AI's role in co-creative processes. The potential for AI models to autonomously critique and creatively collaborate opens new paths in interactive storytelling and content generation, necessitating continued inquiry into ethical and copyright considerations and the impacts on traditional creative sectors.

Conclusion

In all, this paper demonstrates that careful prompting and interactive engagement with LLMs can facilitate creative processes akin to those exhibited by humans, without undermining the intrinsic human value of narrative art. It highlights the dual capacity of LLMs to both imitate existing literary techniques and introduce novel narrative variations, expanding our understanding of machine-aided creativity.