GaelEval: Benchmarking LLM Performance for Scottish Gaelic

Abstract: Multilingual LLMs often exhibit emergent 'shadow' capabilities in languages without official support, yet their performance on these languages remains uneven and under-measured. This is particularly acute for morphosyntactically rich minority languages such as Scottish Gaelic, where translation benchmarks fail to capture structural competence. We introduce GaelEval, the first multi-dimensional benchmark for Gaelic, comprising: (i) an expert-authored morphosyntactic MCQA task; (ii) a culturally grounded translation benchmark and (iii) a large-scale cultural knowledge Q&A task. Evaluating 19 LLMs against a fluent-speaker human baseline ($n=30$), we find that Gemini 3 Pro Preview achieves $83.3\%$ accuracy on the linguistic task, surpassing the human baseline ($78.1\%$). Proprietary models consistently outperform open-weight systems, and in-language (Gaelic) prompting yields a small but stable advantage (+$2.4\%$). On the cultural task, leading models exceed $90\%$ accuracy, though most systems perform worse under Gaelic prompting and absolute scores are inflated relative to the manual benchmark. Overall, GaelEval reveals that frontier models achieve above-human performance on several dimensions of Gaelic grammar, demonstrates the effect of Gaelic prompting and shows a consistent performance gap favouring proprietary over open-weight models.

• The paper introduces GaelEval, a comprehensive benchmark that evaluates LLM performance in Scottish Gaelic through multi-dimensional tasks with robust human baselines.
• It employs a detailed methodology, including a 120-item MCQA for linguistic competence, 908 transcript-based translation tests, and 1,000 cultural knowledge questions.
• Results show that proprietary models outperform open-weight systems and even achieve above-human accuracy in structured tasks, though challenges remain in idiomatic usage.

GaelEval: A Comprehensive Benchmark for Evaluating LLM Performance in Scottish Gaelic

Introduction

The "GaelEval: Benchmarking LLM Performance for Scottish Gaelic" (2604.02135) paper introduces GaelEval, the most ambitious and methodologically rigorous benchmarking suite for minority language assessment, specifically targeting the complex and morphologically rich domain of Scottish Gaelic ("Gaelic"). Existing evaluation frameworks for LLMs lack coverage for minoritized languages, and, where present, are constrained to translation tasks that fail to capture deep structural or cultural competence. GaelEval overcomes these deficiencies by integrating multiple evaluation modalities, measuring not only translation but also morphosyntactic discrimination and culturally informed factual reasoning, with robust human annotation baselines.

Benchmark Design and Methodology

GaelEval is constructed as a multi-dimensional framework with three orthogonal, yet complementary, core tasks:

1. Linguistic Competence (MCQA): An expert-constructed, 120-item multiple-choice set probes grammatical, idiomatic, and morphosyntactic knowledge spanning 16 categories (e.g., case marking, agreement, relativization, idiomatic registers). Items target edge cases not easily solved by simple mapping from high-resource languages.
2. Cultural Translation: Leveraging 908 parallel English-Gaelic podcast transcripts of high cultural content, translation quality is measured bidirectionally using BLEU and chrF, with extensive quality control, including filtering for true semantic alignment and language ID.
3. Cultural Knowledge (QA): Over 1,000 LLM-generated and human-filtered MCQs probe broad Gaelic cultural knowledge from authentic community-sourced narratives, each question carefully filtered for answerability and self-containment.

Each task is paired with a fluent-speaker human evaluation baseline ($n=30$ for the morphosyntax task), addressing the absence of human anchor points in prior minority language benchmarks.

Evaluation Suite: Model Landscape

GaelEval assesses 19 contemporary LLMs, purposefully spanning proprietary and open-weight systems:

• Proprietary: Gemini (Gemini 3 Pro Preview, Gemini 3 Flash), GPT-4/5 family (including Mini/Nano variants), Anthropic Claude (Haiku, Opus)
• Open-weight: DeepSeek R1, Llama 4 Maverick, GLM 4.7, GPT-OSS 120B/20B

All models are evaluated under both Gaelic and English prompts to test the impact of prompting language, following prior findings that in-language prompting can interact differently with resource status.

Results and Analysis

Linguistic Competence Task

Proprietary models—specifically Gemini 3 Pro Preview—attain 83.3% accuracy, surpassing the fluent human baseline of 78.1%, a statistically significant margin. This exceeds the best-performing open-weight models by nearly 40 percentage points, highlighting a pronounced gap in Gaelic competence acquisition.

Figure 1: Category-level accuracy breakdown for the human baseline and the top nine LLMs across 16 grammatical categories, visualizing both aggregate performance and areas of divergence.

Granular analysis reveals:

• LLMs outperform humans in codified, formal morphosyntax (e.g., determiners, pronouns, relative clauses, with up to 0.8 vs 0.53 accuracy), indicating effective generalization over structured grammatical domains.
• Humans retain a strong lead in idiomatic and interactional categories (questions/tags, formulaic language), with LLMs lagging by 0.3–0.4, indicating persistent limitations in capturing authentic register and pragmatic nuances.

Further, in-language Gaelic prompting yields a consistent +2.4% average gain, a positive but modest effect, in contrast to English-prompted benchmarks. Notably, "Mini" and "Nano" variants from all major families show a marked performance degradation, analogously to findings in general scaling laws, emphasizing the necessity for high-capacity models in low-resource morphologically complex language scenarios.

Translation Evaluation

Translation results reinforce directionality asymmetry documented in low-resource NLP: scores degrade substantially when translating into Gaelic (en→gd), with Gemini 3 Flash achieving 71.47 BLEU/79.07 chrF, while several models exceed 70 BLEU when translating Gaelic to English (gd→en). Most open-weight models cannot generate valid Gaelic, with some (e.g., GLM 4.7, GPT-OSS 20B) failing to output any properly formatted response.

The divergence between BLEU and chrF metrics indicates that surface form inflectional errors—critical in Gaelic—remain a barrier for most systems, underscoring the limitations of current tokenization strategies and exposure in language modeling pretraining corpora.

Cultural Knowledge QA

Leading models attain >90% accuracy (Gemini 3 Flash, Gemini 3 Pro, GPT-5), but performance is generally inflated on synthetically generated questions compared to the linguistic MCQA. Rank ordering across benchmarks remains highly correlated, suggesting that synthetic QA benchmarks, while less challenging, provide consistent relative signals for system comparison.

A reverse pattern emerges for prompting language: most models perform worse under Gaelic prompting (mean drop -9.55%), especially for open-weight models, which collapse to near chance under in-language settings, demonstrating robust resource-driven prompt sensitivity.

Theoretical and Practical Implications

The main findings have several clear implications:

• LLM performance in minority and morphologically rich languages is highly non-uniform. Proprietary, high-capacity, multilingual-tuned models (Google/DeepMind’s Gemini) generalize more strongly than open-weight systems, emphasizing the importance of scale, continued data accumulation (especially in translation services), and domain adaptation.
• Above-human performance is now attainable in controlled morphosyntax for minority languages by top-tier LLMs. This raises profound questions for educational technology, CALL, and computational linguistics, but calls for caution: formal grammar competence does not confer full communicative proficiency.
• Weaknesses persist in idiomatic, conversational, and pragmatic contexts. Human participants outperform LLMs in interactive usage and formulaic expressions, aligning with sociolinguistic expectations about what is difficult to acquire outside immersion.
• Prompt language selection interacts with resource status and task type. For complex morphosyntax, in-language prompts still confer an advantage, while for cultural knowledge and knowledge-based tasks, defaulting to English maximizes performance—except for only the strongest systems.
• Synthetic benchmarks offer practical scalability, but inflate absolute scores; careful human validation remains necessary for high-stakes usage and claims regarding real-world parity.

Limitations and Future Directions

Key limitations include a small, non-representative human baseline, a relatively short and unbalanced MCQA instrument, and some LLM-generated datasets with limited expert review. Additionally, one of the evaluated models (GPT-5.2) contributed to data generation, risking self-preference bias.

Among suggested future directions are expansion to larger, more balanced MCQA datasets, increased manual validation of synthetic benchmarks, and further exploration of sociolinguistic and pragmatic appropriateness in conversational AI. Cross-lingual generalization and fine-grained error typology remain open research avenues, particularly around morphologically complex, contact-affected varieties.

Conclusion

GaelEval establishes a new gold standard for minority language evaluation, demonstrating that state-of-the-art LLMs can now outperform fluent humans on selected linguistic tasks while still manifesting large disparities across models and task types. The suite’s categorical breakdown, prompt effect analysis, and synthetic benchmark validation provide a durable protocol for future evaluation in Gaelic and other low-resource, morphologically intricate languages. The results point to both the rapidly advancing state of frontier models and to enduring challenges in communicative and idiomatic mastery, especially by smaller or open-weight LLMs. For AI deployment in minority language contexts, rigorous, language-aware evaluation frameworks such as GaelEval are now indispensable.