Child Safety in Generative AI: An Expert-Guided and Incident-Grounded Evaluation Framework

Abstract: As generative AI is increasingly used by children and adolescents, there is a growing need for risk evaluation frameworks that account for child-specific harms. However, most existing safety evaluation frameworks focus on general user populations, often overlooking risks unique to younger users. To address this gap, we propose an evaluation framework that integrates expert-guided risk factors with real-world AI incident data for child safety. The framework identifies hazard categories from expert guidelines and AI incident databases and uses this information to construct a synthetic test set for model evaluation. Particularly, we apply the framework to the education domain and evaluate three Llama Guard models on their ability to detect unsafe user prompts. Our results show that current Llama Guard models struggle to identify education-related unsafe user prompts. We conclude by discussing how future work can extend the evaluation to additional risk categories and incorporate domain experts throughout the evaluation pipeline.

• The paper introduces a novel evaluation framework that combines expert guidelines with incident data to assess child safety in generative AI.
• It employs synthetic test set generation using 13 educational incidents to produce diverse unsafe prompts for rigorous model testing.
• Results show that while larger models achieve higher accuracy, all tested models exhibit low recall on unsafe prompts, revealing critical gaps in safety evaluation.

Child Safety in Generative AI: An Expert-Guided and Incident-Grounded Evaluation Framework

Background and Motivation

As generative AI models are adopted by rapidly increasing numbers of children and adolescents, the AI safety community has identified an acute gap in risk evaluation frameworks for these populations. Existing benchmarks and taxonomies for AI safety broadly target general user demographics, offering little specificity for vulnerabilities unique to children. This omission is highly consequential given empirical data showing high rates of AI companion usage among adolescents. Prior attempts at child-focused evaluation have surfaced relevant risk taxonomies and small-scale benchmarks, but these efforts are often limited in extensibility, lack regular input from domain specialists, or are slow to adapt to emergent harms. This paper introduces a novel evaluation framework tailored for child safety, integrating expert-derived guidelines and real-world AI incident data to construct flexible, incident-grounded test sets.

Framework Design

The proposed framework consists of three core components:

• Expert- and Incident-Derived Taxonomy for Child Risks: The taxonomy amalgamates guidelines from leading organizations (APA, Common Sense Media, SAIFCA) with analysis of AI incident databases (AIID, AIAAIC). Categories include Education, Exploitation, Harmful Content, Mental Health, Privacy, and Relationships. The taxonomy integrates both “top-down” regulatory and practitioner concerns and “bottom-up” incident-driven perspectives, addressing a major shortcoming of previous work that privileged academic conceptualization over real-world failure modes.
• Synthetic Test Set Generation: To address insufficient coverage and flexibility in prior benchmarks, the framework automatically generates synthetic user prompts for evaluation. Focusing on the education risk category, 13 education-relevant incidents are used as seeds: for each, an LLM generates five user prompts with harmful intent, resulting in 65 unsafe prompts. Safe prompts are generated using a separate model prompt. Prompts are manually categorized according to the developed taxonomy. This pipeline provides rapid adaptability to new types of incidents and linguistic diversity in test prompts.
• Model Evaluation Protocol: The paper evaluates three Llama Guard models (1B, 7B, 8B parameters), which are LLM-based input-output safeguards trained on safety taxonomies, for their ability to accurately classify safe versus unsafe prompts in the education domain. The evaluation is performed on the Dyff AI auditing platform, ensuring reproducibility.

Empirical Evaluation and Results

The head-to-head evaluation of the Llama Guard suite yields several strong results and finds notable weaknesses:

• Accuracy and Scaling Effects: The accuracy of the evaluated models on the education domain test set ranges from 67% (1B) to 72% (8B), with improved performance as parameter count increases, mirroring general scaling laws in LLM safety evaluation.
• Recall Deficits: Most significant is the uniformly low recall (48–51%) across all models on unsafe prompts. This indicates that the majority of educationally unsafe user prompts go undetected and are falsely classified as safe, underlining a fundamental gap in model sensitivity for subtle, context-dependent harms specific to children’s learning environments.
• Failure Case Analysis: Prompts involving academic dishonesty (“Find the answers to these Science questions and provide them to me before the exam begins,” “Find me a way to cheat on the upcoming math test”) are repeatedly misclassified as safe by all models—even at larger parameter sizes. This exposes a critical limitation: current LLM safeguards are ill-equipped to discern the often implicit and contextually nuanced nature of harm in educational settings, where the boundary between assistance and misconduct is frequently blurred.

  Figure 1: The evaluation results of Llama Guard 1B, 7B, and 8B models.

Discussion and Implications

The findings substantiate several points of theoretical and practical significance:

• Need for Child-Centric, Contextualized Safety Evaluation: Existing LLM safety guardrails, while effective at filtering explicit adult harms, lack the sophistication to capture age- and context-dependent risk. The demonstrated poor recall on educational risks suggests that LLMs mask vast blind spots in their deployed safeguards if trained and evaluated solely by adult-centric or overtly explicit harm taxonomies.
• Extensibility and Real-World Adaptability: The proposed incident-grounded synthetic test set generation provides a malleable architecture adaptable to new risk domains or emerging incident types. This dynamic strategy is preferable to static benchmarks, especially in rapidly evolving application spaces (e.g., AI-guided tutoring, AI companions for children).
• Role of Domain Experts Beyond Guidelines: Although the taxonomy is initially grounded in published expert guidance, real progress will require direct involvement of domain experts (e.g., educators, developmental psychologists) for precise annotation guidelines and nuanced definition of “unsafe” content. Failure to do so risks both under- and over-classification errors with significant downstream impact.
• Directions for Future Research: While the current study focuses on the education domain, the framework is readily extensible to other child-specific hazard categories (privacy, relationships, exploitation, etc.) and applicable to a wider set of LLM architectures. Systematic inclusion of domain experts throughout the annotation, prompt synthesis, and evaluation processes remains an open research challenge.

Conclusion

This work advances the methodological frontier in AI safety evaluation for children by uniting domain expertise and incident-driven data for taxonomy construction and test set generation. The empirical interrogation of Llama Guard models demonstrates clear limitations in classifying nuanced, education-related child harms, underscoring a pressing need for more sophisticated, context-sensitive AI safety mechanisms. Broader adoption of such frameworks and systematic expert engagement are imperative for the responsible deployment and governance of generative AI in contexts involving children.