Smaller Large Language Models Can Do Moral Self-Correction

Abstract: Self-correction is one of the most amazing emerging capabilities of LLMs, enabling LLMs to self-modify an inappropriate output given a natural language feedback which describes the problems of that output. Moral self-correction is a post-hoc approach correcting unethical generations without requiring a gradient update, making it both computationally lightweight and capable of preserving the language modeling ability. Previous works have shown that LLMs can self-debias, and it has been reported that small models, i.e., those with less than 22B parameters, are not capable of moral self-correction. However, there is no direct proof as to why such smaller models fall short of moral self-correction, though previous research hypothesizes that larger models are skilled in following instructions and understanding abstract social norms. In this paper, we empirically validate this hypothesis in the context of social stereotyping, through meticulous prompting. Our experimental results indicate that (i) surprisingly, 3.8B LLMs with proper safety alignment fine-tuning can achieve very good moral self-correction performance, highlighting the significant effects of safety alignment; and (ii) small LLMs are indeed weaker than larger-scale models in terms of comprehending social norms and self-explanation through CoT, but all scales of LLMs show bad self-correction performance given unethical instructions.

• The paper demonstrates that small LLMs, with as few as 3.8 billion parameters, can perform moral self-correction when fine-tuned with safety alignment techniques.
• The paper employs methodological prompting and Chain-of-Thought reasoning to evaluate and enhance the models' compliance with ethical norms.
• The paper shows that safety alignment enables resource-efficient small models to achieve ethical performance comparable to or better than larger models.

A Comprehensive Overview of "Smaller LLMs Can Do Moral Self-Correction"

The paper "Smaller LLMs Can Do Moral Self-Correction" investigates the moral self-correction capability of LLMs with less than 22 billion parameters. While it has been noted in the literature that smaller LLMs appear insufficient for moral self-correction, this research provides an empirical validation of this capability in smaller models through fine-tuned safety alignment.

Key Contributions and Findings

The authors challenge the prevailing assumption that models smaller than 22 billion parameters are inept at moral self-correction. Through methodological prompting, they reveal significant findings:

1. Model Scale and Moral Self-Correction: Contrary to earlier beliefs, the study shows that models with as few as 3.8 billion parameters can execute moral self-correction when appropriately fine-tuned with safety alignment techniques. This indicates the substantial role of safety alignment in enhancing moral self-correction without compromising the intrinsic language modeling abilities.
2. Instruction Following and Recognition of Norms: The research explores the ability of small LLMs to understand abstract social norms, follow instructions, and explain decisions in a Chain-of-Thought (CoT) manner. Tests conducted using prompts structured around specificity, negation, and CoT demonstrate that smaller models can indeed comprehend and act upon ethical instructions, albeit with lower effectiveness than larger models.
3. Effectiveness of Safety Alignment: The study empirically validates that safety-aligned small LLMs, notably the phi-3 3.8B model, outperform some larger models when subjected to ethical decision-making tasks. The findings propose a model size threshold for the moral self-correction capability around 3.8 billion parameters, primarily facilitated by safety alignment.

Experimental Framework

The experimental design deploys a variety of LLM scales, including GPT-2, OLMo, Phi-3, and Llama-2, across a spectrum from 355 million to 70 billion parameters. Evaluation metrics are conducted using well-established benchmarks like Winogender for gender bias and BBQ for multiple forms of bias, each assessing different dimensions of bias and ethical reasoning.

The authors apply quantization techniques to enhance computational efficiency, especially with larger models, indicating that even optimized smaller models can perform ethically salient tasks with efficacy under aligned conditions.

Implications and Future Directions

The findings of this study bear considerable implications for both theoretical exploration and practical applications:

• Theoretical Insights: This research advances the understanding of model scalability in alignment with ethical instructions, offering a nuanced perspective that challenges the conception of a linear relationship between model size and moral self-correction capacity.
• Practical Applications: For applications requiring ethical interaction, such as dialogue systems and decision support tools, smaller LLMs provide a more resource-efficient alternative to larger models, given proper safety alignment.
• Future Research Directions: The manuscript suggests future inquiries might investigate the varying behavioral dynamics of LLMs across tasks when confronted with unethical instructions. Furthermore, this study implies that the augmentation of ethical alignment techniques could significantly enhance smaller models' moral reasoning.

Conclusion

The study "Smaller LLMs Can Do Moral Self-Correction" illuminates the overlooked potential of smaller LLMs in moral self-correction under the guidance of safety alignment methods. By challenging preconceived notions about scale and effectiveness, it opens prospects for more resource-efficient deployment of LLMs with ethical awareness, advocating continued research into optimizing alignment methodologies across different model sizes.