Improving Pacing in Long-Form Story Planning

Abstract: Existing LLM-based systems for writing long-form stories or story outlines frequently suffer from unnatural pacing, whether glossing over important events or over-elaborating on insignificant details, resulting in a jarring experience for the reader. We propose a CONCrete Outline ConTrol (CONCOCT) system to improve pacing when automatically generating story outlines. We first train a concreteness evaluator to judge which of two events is more concrete (low-level-detailed). This evaluator can then be used to control pacing in hierarchical outline generation; in this work, we explore a vaguest-first expansion procedure that aims for uniform pacing. We further use the evaluator to filter new outline items based on predicted concreteness. Compared to a baseline hierarchical outline generator, humans judge CONCOCT's pacing to be more consistent over 57% of the time across multiple outline lengths; the gains also translate to downstream stories. All code, data, and models are open-sourced.

• The paper introduces Concoct, which uses a concreteness evaluator to enforce consistent pacing in hierarchical story outline generation.
• It employs a vaguest-first expansion strategy to prioritize expanding the most underdeveloped narrative elements and improve downstream story generation.
• Human evaluations indicate that Concoct enhances pacing without sacrificing coherence or interest, with preferences in about 60% of cases.

Improving Pacing in Long-Form Story Planning

Introduction

The paper "Improving Pacing in Long-Form Story Planning" (2311.04459) addresses an important issue in using LLMs for automatic story outline generation—namely, the challenge of preserving natural pacing. Existing systems often fail in this regard, either glossing over crucial plot points or hyper-focusing on less significant details. Such unnatural pacing can lead to a disjointed narrative experience. To combat this, the authors propose a Concrete Outline Control (Concoct) system that leverages a trained concreteness evaluator designed to enforce consistent pacing during hierarchical outline generation.

Concrete Outline Control System

Concreteness Evaluator

The Concoct system starts by training a concreteness evaluator that judges which of two events is more concrete, based on their level of detail. This evaluator is trained using a curated dataset named Gpt-BookSum, which consists of summaries of story passages varying in granularity. The training process involves comparing pairs of summaries written in a uniform style using ChatGPT, ensuring the model focuses on concreteness rather than stylistic differences.

Figure 1: Concreteness evaluator training. Raw texts are chunked into chapters or passages and summarized using ChatGPT. Summaries are then paired and truncated so that training pairs have similar topic and length.

The concreteness evaluator outputs the probability that one text is more concrete than another, allowing it to facilitate pacing control by hierarchical generation.

Outline Generation

Concoct implements a unique vaguest-first expansion strategy in generating story outlines, utilizing the concreteness evaluator to determine which parts of the outline to expand. By expanding the vaguest nodes first, the system aims to achieve uniform pacing throughout the story.

Figure 2: Stylized example of an outline expansion step. Among all leaf nodes, we select the node which is vaguest according to our concreteness evaluator. We then generate child events for the selected node, filter for concreteness, and finally insert back into the outline.

Downstream Story Generation

While the primary focus is on outline generation, the Concoct system shows improved pacing in downstream story generation as well. It integrates with pre-existing systems to convert outlines into full stories. This involves maintaining passage length consistency to reflect the pacing improvements established at the outline level.

Experimental Evaluation

Human evaluations conducted as part of the research reveal that stories generated from Concoct's outlines are consistently judged to have better pacing compared to baseline methods. In about 60% of cases, Concoct-produced outlines were preferred for their pacing, demonstrating the effectiveness of using the concreteness evaluator for this purpose.

Non-Pacing Error Analysis

Critically, the research ensures that Concoct does not sacrifice other narrative qualities such as coherence and interest. Evaluations show no significant difference in these metrics between Concoct-generated content and baseline stories, indicating that pacing improvements are achieved without adverse effects.

Conclusion

The proposed Concrete Outline Control system represents a significant step forward in addressing pacing issues in story generation by leveraging the concreteness evaluator for outline expansion. While Concoct effectively standardizes pacing, the paper recognizes that storytelling sometimes intentionally varies pacing for effect. Therefore, future work may explore more nuanced pacing strategies, considering narrative engagement beyond uniform pacing. The implications of this research are vast for both AI-generated literature and computational creativity at large, promising more natural and engaging automated storytelling experiences.