A Survey on LLM Inference-Time Self-Improvement

Abstract: Techniques that enhance inference through increased computation at test-time have recently gained attention. In this survey, we investigate the current state of LLM Inference-Time Self-Improvement from three different perspectives: Independent Self-improvement, focusing on enhancements via decoding or sampling methods; Context-Aware Self-Improvement, leveraging additional context or datastore; and Model-Aided Self-Improvement, achieving improvement through model collaboration. We provide a comprehensive review of recent relevant studies, contribute an in-depth taxonomy, and discuss challenges and limitations, offering insights for future research.

• The paper presents a comprehensive survey categorizing inference-time self-improvement methods into independent, context-aware, and model-aided approaches.
• It examines various decoding strategies, including constrained, contrastive, and sampling-based methods, to enhance output quality and model efficiency.
• The study highlights challenges and future directions in maintaining model accuracy without altering original LLM parameters or incurring extra training costs.

Overview of "A Survey on LLM Inference-Time Self-Improvement"

The paper provides a detailed survey on the current state of LLMs with a focus on inference-time self-improvement methods. It categorizes these methods into three distinct groups: Independent Self-Improvement, Context-Aware Self-Improvement, and Model-Aided Self-Improvement. Each category encompasses a variety of strategies designed to enhance model performance and efficiency during the inference phase without altering the original LLM parameters or conducting additional training.

Independent Self-Improvement

This category involves techniques that modify the decoding process to improve LLM performance while retaining the model's frozen parameters intact. The methods are divided into several subcategories, each focusing on different aspects:

1. Constrained Decoding: Utilizes both hard and soft constraints to guide token generation, enabling models to incorporate specific words or modification strategies to improve output quality.
2. Contrastive Decoding: Adjusts token probability based on comparison across conditions, aiming to reduce hallucinations and improve factual correctness.
3. Minimum Bayes-Risk (MBR) Decoding: Prioritizes generating hypotheses that maximize expected utility rather than the most probable single prediction.
4. Parallel Decoding: Accelerates the generation process by producing multiple tokens simultaneously, enhancing efficiency.
5. Sampling-based Decoding: Introduces diversity through stochastic methods, which improve open-ended text generation and reasoning tasks.
6. Tree-Search-based Decoding and Model-level Decoding: These methods mimic logical reasoning processes by exploring multiple potential future states and using intermediate layers to enhance predictions.

Context-Aware Self-Improvement

Methods in this category leverage external information or scenarios to enhance model responses:

1. Prompting with Contextual Information: Uses crafted prompts to enable more accurate few-shot or zero-shot learning.
2. Disturbed Prompt: Involves contrasting probability distributions obtained from regular and specially-crafted prompts to enhance response fidelity.
3. Retrieval-Based Methods: Employ external data collections or retrieval systems to supplement and refine the generated content, enhancing reliability and performance across various tasks.

Model-Aided Self-Improvement

This set comprises approaches that work in concert with smaller auxiliary models or datasets to improve performance:

1. Expert and/or Anti-Expert Models: Decode text by incorporating logits or probability adjustments from specially trained models.
2. Draft Model in Speculative Decoding: Accelerates inference by using smaller models to draft multiple generation paths, which are then verified by the larger LLM.
3. Small LM or Amateur Models: These models assist in various classification and generative tasks, often ensuring alignment with desired output characteristics.
4. Reward Model and Tool/API Utilization: Incorporate external evaluations and specialized tools to guide and constrain generation, ensuring alignment with external benchmarks or constraints.

Implications and Future Directions

The survey emphasizes the importance of inference-time methods in effectively expanding the capabilities of LLMs. These methods offer ways to maintain or even enhance performance without incurring additional training costs or requiring modifications to model architecture. While promising, the paper also identifies several challenges, such as the need for ongoing maintenance when using external data sources or models and computational trade-offs with expansive sampling methods.

Furthermore, as LLMs continue to integrate into various applications, the need for methods that ensure alignment, accuracy, and ethical considerations grows. The paper highlights that future research must address biases inherent in training data and systematic evaluations to ensure LLM reliability and safety in sensitive domains.

In conclusion, the work presented in this paper underscores the versatility and potential of inference-time self-improvement methods to address both practical and theoretical challenges in the deployment of LLMs. Such methods represent a significant step toward realizing more adaptable, efficient, and responsible AI systems.