Cultural Value Differences of LLMs: Prompt, Language, and Model Size

Abstract: Our study aims to identify behavior patterns in cultural values exhibited by LLMs. The studied variants include question ordering, prompting language, and model size. Our experiments reveal that each tested LLM can efficiently behave with different cultural values. More interestingly: (i) LLMs exhibit relatively consistent cultural values when presented with prompts in a single language. (ii) The prompting language e.g., Chinese or English, can influence the expression of cultural values. The same question can elicit divergent cultural values when the same LLM is queried in a different language. (iii) Differences in sizes of the same model (e.g., Llama2-7B vs 13B vs 70B) have a more significant impact on their demonstrated cultural values than model differences (e.g., Llama2 vs Mixtral). Our experiments reveal that query language and model size of LLM are the main factors resulting in cultural value differences.

• The paper shows that prompt variations yield consistent cultural value expressions, though shuffling options introduces measurable divergence.
• The paper finds that language differences critically affect LLM outputs, with t-SNE visualizations and correlation metrics highlighting cultural bias.
• The paper demonstrates that larger model sizes correlate with improved consistency and cultural alignment, despite persistent cross-linguistic disparities.

Cultural Value Differences of LLMs: Prompt, Language, and Model Size

The paper "Cultural Value Differences of LLMs: Prompt, Language, and Model Size" by Qishuai Zhong, Yike Yun, and Aixin Sun provides a comprehensive study on how LLMs express cultural values. It specifically explores the impact of different prompts, languages, and model sizes on these expressions, utilizing Hofstede's latest Value Survey Module (VSM) as the primary assessment tool. The experiment encompasses a variety of model setups and evaluates the cultural bias and consistency in the models' responses.

Key Findings

1. Prompt Variants and Consistency (RQ1): The study finds that variations in prompts within a single language result in relatively consistent cultural values expressed by LLMs. However, models exhibit sensitivity to selection bias induced by shuffling options within prompts. This is evidenced by lower correlation coefficients and distinctive clustering metrics (e.g., DBI and $SS_h$), indicating that while context changes (simulated identities) had little impact, shuffling options led to measurable divergence in responses.
2. Language as a Major Factor (RQ2): Language differences significantly influence the cultural values expressed by LLMs. This is demonstrated through lower correlation coefficients and higher $SS_h$ values when models are queried in different languages. The t-SNE visualizations further underscore this finding, showing substantial separations in responses when the same questions are posed in English versus Chinese. This suggests that multilingual training data introduces distinct cultural biases, supporting the hypothesis that language inherently carries cultural connotations influencing LLM outputs.
3. Model Size and Performance (RQ3): Variations in model size impact the expression of cultural values, with larger models within the same family showing better alignment and consistency. This correlation between model size and proficiency underscores the role of model capabilities in handling complex patterns and context understanding. The study aligns these findings with MMLU scores, indicating that higher performance in language understanding tasks correlates with more coherent cultural value expressions. However, disparities remain, especially when comparing models cross-linguistically, further emphasizing the dominance of language over model size.

Implications and Future Directions

The findings have practical and theoretical implications:

1. Practical Implications: The study highlights the need for careful consideration of prompt engineering to minimize biases in LLM responses. Moreover, developers should be aware of the potential cultural biases introduced by multilingual training data and take steps to mitigate these biases in applications involving cross-cultural interactions.
2. Theoretical Implications: The research confirms the Sapir-Whorf hypothesis in the context of LLMs, illustrating that language structure significantly influences LLM behavior. This opens avenues for exploring linguistic theories within artificial intelligence and further understanding how language-specific training data contributes to cultural bias in models.

Future Developments

Future research should extend the evaluation pipeline to cover a broader range of cultural surveys and include more diverse LLMs to validate the findings. Additionally, integrating user feedback on how language-induced value differences impact end-users can provide deeper insights into mitigating negative effects. The exploration of LLMs with extensive contextual training and continuous refinement of evaluation mechanisms will aid in achieving more accurate and culturally aware AI systems.

In conclusion, the study by Zhong et al. provides a meticulous examination of the cultural values expressed by LLMs. It underscores the critical impact of language and model size and sets the stage for future research into fostering more culturally neutral AI systems.