Local and Global Decoding in Text Generation

Abstract: Text generation, a key component in applications such as dialogue systems, relies on decoding algorithms that sample strings from a LLM distribution. Traditional methods, such as top-$k$ and top-$\pi$, apply local normalisation to the model's output distribution, which can distort it. In this paper, we investigate the effect of this distortion by introducing globally-normalised versions of these decoding methods. Additionally, we propose an independent Metropolis-Hastings algorithm to approximate sampling from globally-normalised distributions without explicitly computing them. Our empirical analysis compares the performance of local and global normalisation across two decoding algorithms (top-$k$ and top-$\pi$) with various hyperparameters, using Pythia LLMs. Results show that, in most configurations, global decoding performs worse than the local decoding version of the same algorithms -- despite preserving the distribution's integrity. Our results suggest that distortion is an important feature of local decoding algorithms.

• The paper demonstrates that global normalization avoids the distortions of local methods but may reduce text generation quality.
• The authors adapt the independent Metropolis-Hastings algorithm to efficiently sample from globally normalized distributions.
• Empirical analysis with Pythia models reveals that the local distortion introduced by normalization can enhance text coherence.

Local and Global Decoding in Text Generation

This paper investigates the effects of local and global normalization on text generation algorithms, with a focus on the widely-used top-$k$ and top-$p$ (or nucleus) decoding methods. The authors introduce globally-normalized variants of these algorithms, enabling a comparison with their traditionally locally-normalized counterparts. A key contribution is the adaptation of the independent Metropolis-Hastings (IMH) algorithm to approximate sampling from globally-normalized distributions.

Key Contributions

1. Global vs. Local Normalization: The study proposes globally-normalized versions of the top-$k$ and top-$p$ decoding algorithms that avoid the distortion caused by local normalization. This allows for an analysis of the impact of such distortions on text generation quality.
2. Metropolis-Hastings Adaptation: The authors adapt the IMH algorithm to sample from the theoretically superior globally-normalized distribution without explicit computation. This provides a practical way of evaluating global normalization despite its computational intractability.
3. Empirical Analysis: The paper provides empirical results comparing local and global versions of top-$k$ and top-$p$ decoding using the Pythia LLMs. The configurations examined range significantly in hyperparameter values for both decoding strategies.

Empirical Findings

• Performance of Global Decoding: In most cases, globally-normalized decoding resulted in worse performance than local normalization, as measured by MAUVE scores. The globally-normalized methods often produced shorter and more repetitive sequences, suggesting that distortion in local decoding might inadvertently improve text quality.
• Distortion's Role: Surprisingly, the distortion introduced by local normalization appears to enhance performance by promoting longer and more coherent text. The paper argues that this distortion, often perceived as a drawback, may actually be beneficial for certain tasks.
• Sampling Quality: By using IMH, the sampling from globally normalized distributions approximated the true distribution effectively when a sufficient number of iterations were used. However, the additional computational cost and complexity may offset the theoretical advantages.

Implications and Future Directions

This work raises important questions about the role of normalization in text generation. The results suggest that local normalization, although distorting the distribution, could be contributing positively to the quality of generated text. This counters the intuitive preference for distribution-preserving global normalization.

Theoretical Implications: The findings challenge the assumption that distribution integrity (i.e., global normalization) inherently results in better text generation. The bounds provided on the KL divergence between the two approaches elucidate the potential for significant distributional differences, prompting a reevaluation of distortion's role in practical applications.

Practical Implications: In practice, developers of dialogue systems and other generative AI applications should consider the possibility that local distortions may improve generation quality. These findings may drive the development of new decoding algorithms that purposefully leverage distortion.

Future Research: Further studies should explore the balance between probability distribution fidelity and generation quality across different model architectures and languages. This paper also opens avenues for extending these analyses to other tasks, such as machine translation and summarization, where different trade-offs might apply.

In sum, this paper provides significant insights into decoding strategies' role in text generation, inviting researchers to reexamine the assumptions underlying current practices and encouraging further exploration of the nuanced effects of normalization techniques in AI.