Do Stop Me Now: Detecting Boilerplate Responses with a Single Iteration

Abstract: LLMs often expend significant computational resources generating boilerplate responses, such as refusals, simple acknowledgements and casual greetings, which adds unnecessary cost and latency. To address this inefficiency, we propose a simple yet highly effective method for detecting such responses after only a single generation step. We demonstrate that the log-probability distribution of the first generated token serves as a powerful signal for classifying the nature of the entire subsequent response. Our experiments, conducted across a diverse range of small, large, and reasoning-specialized models, show that the first-token log-probability vectors form distinctly separable clusters for different response types. Using a lightweight k-NN classifier, we achieve high accuracy in predicting whether a response will be a substantive answer or a form of boilerplate response, including user-specified refusals. The primary implication is a practical, computationally trivial technique, optimizing LLM inference by enabling early termination or redirection to a smaller model, thereby yielding significant savings in computational cost. This work presents a direct path toward more efficient and sustainable LLM deployment.

• The paper demonstrates that analyzing first-token log-probabilities enables near-perfect classification of boilerplate responses.
• The k-NN classifier, applied across various model architectures, achieves accuracy above 0.98 with minimal computational overhead.
• The approach offers practical benefits for LLM deployment by enabling early termination and efficient model routing to reduce latency and cost.

Efficient Detection of Boilerplate Responses in LLMs via First-Token Log-Probabilities

Introduction

The paper presents a method for early detection of boilerplate responses—such as refusals, acknowledgements, and greetings—in LLMs by analyzing the log-probability distribution of the first generated token. This approach enables computationally trivial classification of response types, facilitating early termination or model routing to optimize inference cost and latency. The method is validated across a spectrum of model sizes and architectures, demonstrating robust separability of response classes and high classification accuracy with a simple k-NN classifier.

Methodology

The core hypothesis is that the log-probabilities assigned to the first token during LLM generation encode sufficient information to distinguish between substantive and boilerplate responses. The authors construct a dataset of 3,000 chats spanning four classes: Refusal, Thanks, Hello, and Chat. For each prompt, the log-probability vector of the first generated token is extracted and used as the feature representation.

A k-Nearest Neighbors (k-NN) classifier ($k=3$) is trained on these vectors for each model. The separability of response types is visualized using 2D t-SNE projections, revealing distinct clusters corresponding to each class.

Figure 1: 2D T-SNE plot of first-token log-probabilities for Small LLMs (Llama 3.2 3B, Qwen 2.5 1.5B, Gemma 3 1B).

The method is further tested for system-prompt-induced refusals, confirming that the first-token log-probabilities shift toward the Refusal cluster when the system prompt instructs the model to refuse a request.

Figure 2: 2D T-SNE plot of first-token log-probabilities for Small LLMs, including the Cake Recipe experiment. The black circle: recipe request; black cross: refusal due to system prompt.

For reasoning-specialized models, the authors introduce an empty-thinking phase to isolate the first substantive token, maintaining cluster separability.

Figure 3: 2D T-SNE plot of post-empty thinking first-token partial log-probabilities for Reasoning Models (DeepSeek-R1 8B, Phi-4 Reasoning Plus).

For large, cloud-based models (GPT-4o, Gemini 2.0 Flash), only the top-20 token log-probabilities are available, yet the method remains effective.

Figure 4: 2D T-SNE plot of first-token partial log-probabilities for LLMs (GPT-4o, Gemini-2.0 Flash).

Experimental Results

The k-NN classifier achieves near-perfect accuracy across all tested models:

• Small LMs: Accuracy $>$ 0.99, F1 $>$ 0.99 for all classes.
• Reasoning LMs: Accuracy $>$ 0.99, F1 $>$ 0.99.
• Large LMs: Accuracy $\approx$ 0.98, F1 $\approx$ 0.94 (GPT-4o), $\approx$ 0.88 (Gemini 2.0 Flash).

The method robustly identifies refusals triggered by both internal safeguards and explicit system prompts. Notably, the classifier distinguishes between incapability-induced refusals (e.g., missing context) and those due to policy constraints.

Implementation Considerations

• Computational Overhead: The approach requires only a single forward pass and extraction of the first-token log-probabilities, incurring negligible additional cost.
• Model-Agnosticity: The method generalizes across architectures and scales, though log-probability spaces are not directly comparable between models.
• API Limitations: For closed-source APIs, partial log-probability vectors suffice for effective classification.
• Deployment: The classifier can be integrated into LLM serving stacks to enable early termination, model routing, or semantic caching for boilerplate queries.

Practical and Theoretical Implications

This work provides a direct mechanism for reducing LLM inference cost and latency by preemptively filtering non-substantive responses. The findings suggest that global response attributes are encoded in the initial token distribution, supporting theories of emergent response planning in LLMs. The method complements existing refusal detection and semantic caching strategies, offering a lightweight alternative that does not require retraining or manual token lists.

The approach is extensible to additional boilerplate categories, multi-language settings, and potentially multi-modal models. It also opens avenues for probing other global response characteristics (e.g., factuality, reasoning depth) at early generation stages.

Conclusion

The paper demonstrates that first-token log-probabilities provide a powerful, model-agnostic signal for early detection of boilerplate responses in LLMs. The proposed k-NN-based classification method achieves high accuracy with minimal computational overhead, enabling practical optimizations in LLM deployment. Future research may explore broader response categories, cross-lingual generalization, and integration with multi-modal architectures.