Characterizing Cultural Localization in AI-Generated Stories

Abstract: The global use of artificial intelligence has increased interest in assessing the ability to generate culturally localized content, including stories. Cultural localization in stories often occurs through either templated localization -- the use of cultural markers (e.g., names, locations) in a generic narrative -- or holistic localization -- the variation of plots, values, and themes, in addition to cultural markers. We propose a method to measure the degree to which content was generated through templated localization. Specifically, we identify the lexical tokens that distinguish stories across nationalities and measure the similarity of the narratives that remain after removing them. In stories generated by five models on 125 topics for 193 nationalities, our method is able to detect that only a small subset (9-17%) of the vocabulary accounts for the variation across nationalities and that the narratives that remain after removing them contain repeated multi-word sequences, suggesting the presence of a shared culturally-agnostic narrative template. Finally, we characterize the cultural markers for their stereotypicality and offensiveness, finding that markers from 19 countries, mostly located in the Global South, are on average offensive.

• The paper demonstrates that LLMs primarily rely on a narrow set of cultural markers, resulting in templated localization rather than true narrative adaptation.
• It employs a two-stage methodology using NPMI for lexical marker identification and template homogeneity measures to assess cultural differentiation.
• Empirical results indicate that masking just 9–17% of vocabulary drastically reduces nationality classification accuracy, revealing risks of stereotype propagation and cultural erasure.

Cultural Localization Mechanisms in LLM Story Generation

Conceptual Background and Motivation

The paper "Characterizing Cultural Localization in AI-Generated Stories" (2606.14626) addresses the critical question of whether LLMs generate culturally adapted stories by genuinely varying narrative structure and values (holistic localization) or predominantly by inserting surface-level cultural markers (templated localization). The distinction bears implications for cultural competence, representational diversity, and potential harms such as stereotype propagation and cultural erasure. Previous work has shown substantial lexical variation and presence of culturally salient entities in model outputs, but the underlying mechanisms of localization have remained opaque.

Narrative structure has long been studied in computational linguistics and sociolinguistics as a channel for cultural identity, where plot elements, values, and discourse structures can be representative of cultural traditions [colby_partial_1973, hobson-etal-2024-story]. Separating narrative template from surface realization enables explicit measurement of where and how models localize content.

Methodological Framework

The proposed method adopts a two-stage analytical pipeline:

1. Lexical Marker Identification: For each nationality, the method computes normalized pointwise mutual information (NPMI) for lexical tokens to isolate the minimal set of words whose removal renders stories indistinguishable by a classifier. These are operationalized as candidate cultural markers.
2. Narrative Template Homogeneity: Following marker removal, the similarity across stories (template images) is measured using normalized longest common substring (LCS) and Jaccard over $n$-gram sequences. High similarity post-masking indicates templated localization.

Classifier performance (multi-class, 193-way) is used as the primary evaluation metric; masking the top culturally salient tokens rapidly drops macro F1 to random chance, indicating that cultural differentiation is heavily concentrated in a small fraction of vocabulary.

Figure 1: F1 of nationality classifier as a function of number of masked words, demonstrating rapid collapse of distinguishability upon removal of high NPMI cultural markers.

Empirical Evaluation

The experimental protocol uses five LLMs (GPT-3.5 Turbo, GPT-4o Mini, Llama 3.1 8B Instruct, Llama 3.3 70B Instruct, Gemma 3 12B Instruct) to generate stories for 125 topics and 193 nationalities, totaling over 600,000 generations. Prompts are standardized to elicit narratives for a nationality on a topic.

Key findings:

• Across models, only 9–17% of vocabulary are required to differentiate stories by nationality, as measured via classifier macro F1.
• Masking these markers increases inter-nationality narrative similarity by up to 66% (Jaccard, Gemma 3), with the remaining text sequences exhibiting frequent multi-word repetition and latent culturally-agnostic templates.
• Classifiers trained on randomly masked words do not reach the same rapid collapse in distinguishability, validating marker specificity.
• Many cultural markers overlap with stereotypical or offensive attributes, predominantly for countries in the Global South.

  Figure 2: Nationality classifier macro F1 trajectory for GPT-3.5 Turbo as a function of masked marker fraction, confirming concentration of cultural distinction in lexical markers.

Stereotype Characterization and Representational Harms

Cultural markers are cross-referenced with the SeeGULL stereotype dataset to measure precision and offensiveness. Precision is highest in countries where dominant languages are high-resource and the stereotypes themselves tend toward non-offensive or neutral. However, 19 countries demonstrate positive average offensiveness, mostly in Africa and West Asia, underscoring uneven risks of representational harm.

Examples of markers include explicit stereotypical terms ("ninja" for Japan, "terrorist" for Lebanon, "beggar" for Bangladesh) and attributes ("proper" for Britain, "pushy" for Israel).

Implications and Theoretical Considerations

The paper provides strong evidence that LLMs, despite adaptive prompting, default to templated localization rather than holistic localization for story generation. As a result:

• Narrative homogeneity: Story templates are reused, and meaningful plot, value, or genre variation across cultures is minimal.
• Overestimation of competence: Evaluations measuring presence of cultural markers alone are insufficient and can misrepresent true model capacity for narrative adaptation.
• Potential harms: Homogenization and stereotype propagation disproportionately affect lower-resourced regions and marginalized groups, echoing documented sociotechnical harms [shelby_sociotechnical_2023, qadri2025risksculturalerasurelarge].

The results challenge the notion that end-to-end-trained models inherently diversify narrative structures and instead behave akin to slot-filling template pipelines. This aligns with recent findings on LLM generation homogeneity, epistemic collapse, and diminished plot diversity [xu2024echoesaiquantifyinglack, doshi_generative_2024].

Practical and Future Directions

Practically, the templated localization exposes limitations in current LLM deployments for cross-cultural storytelling, education, and creative writing. Improving cultural competence requires explicit intervention at the narrative structure level—potentially via conditioning on cultural schemata, domain-adapted narrative templates, or fine-grained expert annotation.

Future work should target:

• Multi-turn prompting: Investigating whether iterative, nuanced prompts can break narrative homogeneity.
• Genre and task expansion: Applying the framework beyond children's stories to multimedia, adult fiction, or screenwriting.
• Alternative definitions of culture: Extending analysis to intra-national, linguistic, and community-based identities.
• Community-engaged stereotype benchmarking: Constructing reference sets through participatory methods to mitigate harms.

AI narrative generation research must incorporate multidisciplinary narrative theory and humanistic expertise to assess and optimize cultural adaptation [hamilton_narrabench_2026, dagstuhl:ai-culture].

Conclusion

"Characterizing Cultural Localization in AI-Generated Stories" evidences that current LLMs achieve cross-cultural narrative adaptation predominantly through surface-level lexical markers, with minimal holistic divergence in plot, values, or structure. This templated localization mechanism risks cultural erasure, propagates stereotypes, and invalidates naive evaluations of cultural competence. The analytical approach provides a rigorous foundation for quantitative assessment, and reinforces the necessity of theoretical integration and community participation for equitable and diverse AI storytelling.