Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing

Abstract: Despite their strong performance on many tasks, pre-trained LLMs have been shown to struggle on out-of-distribution compositional generalization. Meanwhile, recent work has shown considerable improvements on many NLP tasks from model scaling. Can scaling up model size also improve compositional generalization in semantic parsing? We evaluate encoder-decoder models up to 11B parameters and decoder-only models up to 540B parameters, and compare model scaling curves for three different methods for applying a pre-trained LLM to a new task: fine-tuning all parameters, prompt tuning, and in-context learning. We observe that fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional generalization in semantic parsing evaluations. In-context learning has positive scaling curves, but is generally outperformed by much smaller fine-tuned models. Prompt-tuning can outperform fine-tuning, suggesting further potential improvements from scaling as it exhibits a more positive scaling curve. Additionally, we identify several error trends that vary with model scale. For example, larger models are generally better at modeling the syntax of the output space, but are also more prone to certain types of overfitting. Overall, our study highlights limitations of current techniques for effectively leveraging model scale for compositional generalization, while our analysis also suggests promising directions for future work.

• The paper reveals that fine-tuning larger models often shows flat or negative scaling trends for compositional generalization.
• The paper demonstrates that prompt tuning leverages increased model scale to yield positive effects on compositional generalization.
• The paper finds that in-context learning benefits from scaling but remains sensitive to the quality of exemplar retrievers compared to fine-tuning.

Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing

The paper "Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing" investigates how increasing the size of pre-trained LLMs affects their capacity for compositional generalization, particularly in semantic parsing tasks. The authors evaluate encoder-decoder models like T5, as well as decoder-only models like PaLM, scaling up to 540 billion parameters. Analyzing the potential of three methods of task adaptation—fine-tuning, prompt tuning, and in-context learning—the study deepens our understanding of scaling's role in enhancing compositional generalization.

Large models, when pre-trained, often excel in learning direct mappings between inputs and outputs within a distribution they were exposed to during training. Yet, the ability to generalize compositions of these learned elements when encountered in novel and out-of-distribution arrangements remains a challenge. While it is known that larger model sizes can lead to improvements across various tasks, whether this scaling extends to compositional generalization has not been fully explored. By systematically evaluating scaling curves on models with varying parameter sizes, the authors probe further into the potential theoretical and practical implications.

The study identifies the performance trends for each task adaptation method:

1. Fine-Tuning: When fine-tuning all model parameters, larger models did not consistently show improved compositional generalization. In many cases, the scaling curves were either flat or negative, highlighting limitations in current approaches when leveraging model size for this specific aspect of generalization.
2. Prompt Tuning: Positive results were observed with prompt tuning, particularly for larger models. The scaling curves for this method were mostly positive, indicating that prompt tuning could effectively utilize scaling to enhance compositional generalization abilities.
3. In-Context Learning: With larger model scales, in-context learning exhibits positive scaling trends. However, it is frequently outperformed by smaller models employing fine-tuning. The method's dependency on the quality and effectiveness of retrievers used for exemplar selection is evident.

Moreover, the paper details error trends, noting that larger models are more adept at modeling output syntax but may also overfit to known distributions when fine-tuned. This propensity hints at a balance larger models must achieve: leveraging size without risking overfitting.

The findings propose significant insights into the operational limits and potential future directions for employing larger LLMs. While scaling remains beneficial in certain cases, the study calls for innovation in methods like enhanced retrievers for in-context learning, alternative output formats, and constrained decoding to avert syntax errors. These areas highlight possible future advancements in AI, specifically within semantic parsing and beyond, to better mimic human-like language understanding.

Overall, this detailed exploration of scaling behavior helps guide ongoing developments in AI, underscoring that parameter count alone is not a panacea for compositional generalization. Future work may move toward hybrid approaches combining scale with improved architectural strategies to circumvent current bottlenecks.