Why are all LLMs Obsessed with Japanese Culture? On the Hidden Cultural and Regional Biases of LLMs

Abstract: LLMs have been showing limitations when it comes to cultural coverage and competence, and in some cases show regional biases such as amplifying Western and Anglocentric viewpoints. While there have been works analysing the cultural capabilities of LLMs, there has not been specific work on highlighting LLM regional preferences when it comes to cultural-related questions. In this work, we propose a new dataset based on a comprehensive taxonomy of Culture-Related Open Questions (CROQ). The results show that, contrary to previous cultural bias work, LLMs show a clear tendency towards countries such as Japan. Moveover, our results show that when prompting in languages such as English or other high-resource ones, LLMs tend to provide more diverse outputs and show less inclinations towards answering questions highlighting countries for which the input language is an official language. Finally, we also investigate at which point of LLM training this cultural bias emerges, with our results suggesting that the first clear signs appear after supervised fine-tuning, and not during pre-training.

• The paper introduces the CROQ dataset and evaluation framework to systematically expose hidden cultural biases in multilingual LLMs.
• It reveals that LLMs consistently overemphasize Japan as a cultural reference, even when prompted in languages without explicit regional cues.
• The study shows that supervised fine-tuning and alignment processes amplify cultural biases, underscoring the need for more balanced training data.

Measuring and Explaining Cultural and Regional Biases in LLMs: Insights from the CROQ Framework

Introduction

This paper, "Why are all LLMs Obsessed with Japanese Culture? On the Hidden Cultural and Regional Biases of LLMs" (2604.21751), presents a comprehensive methodology and analysis to systematically surface and measure cultural and regional biases within state-of-the-art LLMs. In contrast to prior work which primarily exposed persistent Western and Anglocentric leanings, this study introduces the Culture-Related Open Questions (CROQ) dataset—a large multilingual set of open-ended, regionally-ambiguous cultural prompts—and probes both closed and open-weight LLMs in 24 languages. The findings challenge received assumptions by uncovering a pronounced and consistent overemphasis on Japan as a cultural reference point across essentially all LLMs, as well as dissecting how language, training data resource levels, and model alignment protocols modulate these biases.

CROQ Dataset and Evaluation Framework

The newly introduced CROQ dataset consists of 31,680 open-ended cultural questions systematically covering 11 major cultural domains and 66 fine-grained subtopics, translated and post-edited across 24 languages. The key innovation is the question pattern, which excludes all explicit regional markers and forces the LLM to implicitly select and ground cultural content within a concretely chosen country or region, thereby surface-latent cultural priors. Model responses are further processed by a secondary LLM (‘judge’) which systematically extracts the referenced regions with high validated accuracy.

Figure 1: The CROQ evaluation pipeline: models answer open-ended cultural questions without explicit regional cues, and a judge model extracts referenced regions, exposing intrinsic cultural priors.

This framework generalizes cultural probing beyond closed-form question-answering by eliciting free-form geographical anchoring, thus permitting a distributional analysis of model biases.

Quantification of LLM Cultural Bias: Main Findings

Language-Region Coupling and Exogenous Biases

Across both closed and open LLMs and the full multilingual set, models predominantly anchor responses to the country co-official with the prompt language (43–78% for major models). However, when responses reference regions outside the language's primary association, there is a robust and model-agnostic overrepresentation of Japan, frequently exceeding the United States, India, and China as top exogenous references. This hierarchy remains stable across almost all languages, including those that are geographically or culturally distant from Japan.

Figure 3: Country references in responses from major models (e.g., Llama-3.1, Gemma-2, Mistral, Qwen2.5) to English questions, highlighting the disproportionate focus on Japan and the USA even for non-English cultures.

Topic-level analysis reveals that while the U.S. may predominate in economic or geographic prompts, Japan remains salient (often first or second) in nearly all domains except for rare topic-specific shifts (e.g., Greece in Beliefs, South Korea in Media & Entertainment). Frequency distributions show that references to other countries sharply diminish, indicating strong mode-collapse on a handful of internationally salient cultures.

Impact of Language Resource Level

Language resource level, as proxied by CommonCrawl coverage, has a statistically significant correlation with output diversity (Spearman’s $r_s = 0.843$, $p < 0.001$): high-resource languages yield answers more frequently referencing exogenous countries (notably Japan and the U.S.) and demonstrate higher entropy across country references. In contrast, models prompted in low-resource languages default to highly self-referential or non-diverse answers, sometimes resulting in non-answers, reinforcing a feedback loop of linguistic marginalization.

Figure 5: Diversity of referenced countries correlates with CommonCrawl coverage and speaker population, indicating that training data resource levels constrain cultural generalization in LLMs.

Emergence of Bias: Pretraining vs. Post-training

The study systematically isolates the origin of cultural regional biases through controlled comparisons of base, supervised fine-tuned (SFT), and instruction-tuned open-weight models. Base models (prior to instruction tuning) exhibit much more balanced distributions of cultural references, with no single dominant exogenous region. However, after SFT and instruction alignment, there is a marked collapse in diversity and entropy: references sharply concentrate on the U.S. and Japan, regardless of model origin or scale.

Figure 7: For Olmo models, the emergence of strong Japan/USA bias occurs after SFT; instruction alignment only marginally modifies the entropy collapsed during fine tuning.

The dominant cultural skew thus arises not during self-supervised pretraining but during SFT and alignment, which is intended to enhance model helpfulness and reduce undesirable biases, but often paradoxically hard-codes new, homogenized cultural stereotypes.

Model Family Variability

Frontier LLMs, both commercial closed models (e.g., GPT-4o, Gemini-2.5, Claude-3.5) and open-weight models (e.g., Llama-3, Gemma-2, Qwen2.5, Magistral, DeepSeek, Command-R, OLMo), all manifest this regional bias profile, but with subtle differences. For example, Command-R produces the most diverse outputs and is least Japan-skewed (sometimes favoring the U.S.), but even here, mode-collapse is evident. Overall, the bias toward Japan is robust across all architectures, degrees of openness, and model scales.

Figure 9: Output distribution (excluding self-references) by both model and prompt language; the salience of Japan (JP) persists independently of linguistic context.

Theoretical and Practical Implications

The findings substantially complicate prevailing narratives regarding LLM cultural bias. The default overemphasis on Japan—distinct from prior Western/American overrepresentation—challenges simple theoretical explanations based on Western training data dominance, suggesting the need to interrogate the structure and balance of instruction datasets and alignment protocols.

From an applied perspective, the results indicate that current LLMs, when used for open-ended cultural tasks (e.g., chatbots, knowledge assistants, educational agents), will default to overfitting a handful of globally-salient, frequently East Asian (Japanese), U.S., and other G7 cultural exemplars, even in low-resource linguistic contexts. This undermines the goal of designing LLMs as culturally robust, locally-contextualized assistants.

Implications for Future LLM Development

These results call for reconsideration of SFT and alignment dataset designs, with targeted modifications to ensure balanced cultural representation and coverage, especially during post-training. The consistent mode collapse toward Japan and similar cultures is likely a reflection of annotation set artifacts and alignment reference biases injected during RLHF and instruction data curation.

Future research should:

• Audit and restructure supervised alignment and instruction sets to ensure proportional representation across a wider array of cultures, especially beyond the usual Japan-U.S.-Western Europe core.
• Develop mechanisms for explicit balancing during RLHF to regularize references across global cultures and minimize reinforcement of redundant exemplars.
• Extend cross-lingual alignment to avoid language-resource-driven mode collapse in low-resource languages.
• Rigorously evaluate the impact of cultural referencing biases in applied downstream tasks (e.g., cross-cultural dialog, international knowledge retrieval, multi-region personalization).

Conclusion

Through the introduction of CROQ and systematic analysis across languages, models, and training stages, this work establishes that cultural and regional bias in LLMs emerges predominantly during supervised fine-tuning and alignment, manifesting as a form of cross-model mode-collapse on Japan and a narrow set of major cultures. The implications expose critical limitations in model cultural coverage, highlight the unintended consequences of current post-training protocols, and offer clear directions for improving the global cultural robustness and epistemic diversity of future multilingual LLMs.