Measuring AI "Slop" in Text

Abstract: AI "slop" is an increasingly popular term used to describe low-quality AI-generated text, but there is currently no agreed upon definition of this term nor a means to measure its occurrence. In this work, we develop a taxonomy of "slop" through interviews with experts in NLP, writing, and philosophy, and propose a set of interpretable dimensions for its assessment in text. Through span-level annotation, we find that binary "slop" judgments are (somewhat) subjective, but such determinations nonetheless correlate with latent dimensions such as coherence and relevance. Our framework can be used to evaluate AI-generated text in both detection and binary preference tasks, potentially offering new insights into the linguistic and stylistic factors that contribute to quality judgments.

• The paper introduces a comprehensive taxonomy of AI 'slop' by synthesizing expert insights and systematic span-level annotations.
• It employs logistic regression and domain-specific analysis to quantify text quality issues in news articles and QA passages.
• The study reveals that current automated and LLM-based methods struggle to capture the nuanced, multi-dimensional aspects of slop.

Measuring and Characterizing AI "Slop" in Text

Introduction

The paper "Measuring AI 'Slop' in Text" (2509.19163) presents a systematic framework for defining, annotating, and measuring the phenomenon of "slop"—a term increasingly used to describe low-quality, generic, or bothersome AI-generated text. The authors develop a taxonomy of slop through expert interviews, operationalize its measurement via span-level annotation, and analyze both human and automatic methods for slop detection. The work addresses the lack of a formal definition and measurement protocol for slop, which is critical given the prevalence of LLM-generated content in writing and information tasks.

Taxonomy and Definition of "Slop"

The authors construct a multi-dimensional taxonomy of slop by synthesizing qualitative input from 19 experts in NLP, writing, linguistics, and philosophy. The taxonomy is organized into three principal axes:

• Information Utility: Density and relevance of information.
• Information Quality: Factuality and bias/subjectivity.
• Style Quality: Structure (repetition, templatedness), coherence, fluency, verbosity, word complexity, and tone.

Each axis is mapped to measurable codes, some of which can be quantified with existing automatic metrics (e.g., token entropy for density, subjectivity lexicons for bias), while others (e.g., relevance, coherence, fluency) require human annotation due to the limitations of current automated approaches.

Figure 1: Sample of annotations over a human-written news article highlighting indicators of ``slop'' (red), human-writing (green), and both (yellow).

Annotation Protocol and Inter-Annotator Agreement

The annotation process involves professional copy-editors labeling spans in news articles and QA passages for slop indicators. Annotators first provide a binary judgment of slop, then highlight word-level spans corresponding to specific codes. Agreement is measured using span-level precision, Cohen's $\kappa$, Gwet's AC$_1$, and Krippendorff's $\alpha_{MASI}$, with results indicating moderate to high agreement on problematic spans but only fair agreement on binary slop judgments. This reflects the inherent subjectivity and construct-level complexity of slop assessment.

Figure 2: Years of experience for each expert, demonstrating the depth of domain expertise in the definition process.

Empirical Analysis of Slop Indicators

Logistic regression models fitted to annotated data reveal that all seven codes are significant predictors of slop judgments, with relevance, density, and tone being the strongest. The analysis is stratified by domain:

• News Articles: Style quality (coherence, tone), information utility (density, relevance), and bias are most predictive.
• QA Passages (MS MARCO): Factuality and structural issues dominate, with less emphasis on density and tone due to shorter passage length.

Figure 3: ``Slop'' codes most predictive of the overall positive label for (a) the entire corpus, (b) news, and (c) MS MARCO.

The domain-specific variation underscores the necessity of contextualizing slop evaluation according to the purpose and genre of the text.

Automatic Measurement and Model-Based Evaluation

The authors evaluate the feasibility of automatic slop detection using linear models trained on available metrics, reward models (WQRM), and LLM-based zero/few-shot prompting. Key findings include:

• Linear Models: Achieve AUPRC scores double the prevalence baseline (0.52 for news, 0.55 for MS MARCO), but fail to fully capture the latent qualities of slop.
• Reward Models (WQRM): Show moderate correlation with slop annotations (0.25 for news, 0.15 for MS MARCO), indicating partial alignment but insufficient coverage of slop dimensions.
• LLMs-as-Judges: Off-the-shelf LLMs (GPT-5, Deepseek-V3, o3-mini) exhibit low agreement with human annotators ($\kappa \sim 0$), under-predict slop, and have poor precision/recall in span extraction tasks.
• Fine-Tuned Span Extraction: Training a Qwen-7B model on augmented data yields improved precision (0.33) and recall (0.22), but overall F1 remains low, highlighting the challenge of automating slop identification.

Practical and Theoretical Implications

The taxonomy and annotation framework provide a foundation for more granular and interpretable evaluation of LLM-generated text, moving beyond surface-level metrics like BLEU/ROUGE. The results demonstrate that slop is a multi-faceted construct, with subjective and latent dimensions that are not adequately captured by current automatic metrics or LLM-based evaluators. This has direct implications for the development of reward models, alignment protocols, and evaluation pipelines in generative NLP.

The findings also suggest that slop is not exclusive to AI-generated text; human writing can exhibit similar deficiencies, though often for different functional reasons (e.g., intermediary drafts). The domain-specific analysis further indicates that quality assessment must be tailored to the context and intended use of the text.

Figure 4: ``Slop'' Evaluation consent form given to annotators prior to the study, ensuring ethical data collection.

Future Directions

The paper identifies several avenues for future research:

• Development of Reliable Automatic Metrics: Especially for relevance, coherence, and fluency, which are currently bottlenecks in automated slop detection.
• Improved LLM Alignment: Reward models should incorporate the multi-dimensional taxonomy to better capture human preferences and avoid overfitting to superficial cues.
• Domain-Specific Evaluation: Further work is needed to adapt slop assessment frameworks to different genres and tasks.
• Human-in-the-Loop Annotation: Given the subjectivity and complexity of slop, hybrid approaches combining human and machine judgments may be necessary for robust evaluation.

Conclusion

This work provides a rigorous operationalization of "slop" in AI-generated text, offering a taxonomy, annotation protocol, and empirical analysis that collectively advance the state of evaluation in generative NLP. The results highlight the limitations of current automatic and LLM-based evaluation methods, emphasizing the need for multi-dimensional, interpretable, and context-sensitive approaches. The taxonomy and annotated datasets released by the authors will serve as valuable resources for future research on text quality, LLM alignment, and the mitigation of low-quality generative outputs.