- The paper presents a runtime-aware annotation framework that reveals hidden, safety-relevant signals in Scratch projects beyond static metadata.
- It employs a hybrid method combining automated runtime event simulation and human adjudication to analyze 500 projects, with 93% requiring dynamic inspection.
- The findings underscore the need for runtime evaluation in content curation, guiding safer educational practices and more robust dataset constructions.
Content Hidden Behind Execution: Runtime Analysis of Public Scratch Projects
Problem Motivation and Context
The paper addresses a core challenge in block-based programming education: the curation and classroom reuse of public Scratch projects, which are not simply static code artifacts but executable, interactive media. The central issue is that safety-relevant or age-inappropriate content may not be detectable through metadata such as title, thumbnail, or description, but instead emerges only during execution, gameplay progression, specific failure states, user interactions, dynamic costume switching, audio events, or concealed triggers. This runtime-dependence complicates content screening for educators, researchers building datasets, and teachers selecting classroom exemplars. The study frames this as a "runtime-revealed sensitive content" problem, arguing that current curation practices relying on static review are insufficient for ensuring age-appropriate, safe learning contexts.
This work is situated at the intersection of multiple threads: constructivist computing education, resource selection and metadata-driven curation, analysis and runtime support for Scratch, youth safety in online environments, and interactive media classification. It extends prior literature in computational thinking assessment [brennanresnick2012ct], automated Scratch code analysis [li2026raven, fraser2021litterbox], and digital resource triage [currier2004metadata, abramovich2013metadata]; but pivots focus from correctness and pedagogical affordances to content appropriateness and dataset/documentation rigor. The approach parallels game rating methodologies (e.g., ESRB, PEGI), which employ runtime evidence to assess user experience rather than surface-level descriptors.
Taxonomy and Annotation Framework
The paper introduces a multi-dimensional, runtime-aware annotation scheme comprising five axes:
- Content Category (C1–C9): Multi-label, capturing violence, horror, gore, sexual/NSFW, hate/discrimination, harassment/bullying, self-harm/suicide, political/societal, and moderation-conflict/report avoidance.
- Risk/Severity Level: 0–4 ordinal, from benign to high-risk and likely policy-violating.
- Evidence Channel (E0–E6): Source of the signal (metadata, thumbnail, static assets, code, passive/interactive runtime, public context).
- Reveal Mechanism (M0–M8): How the signal is surfaced (static, passive runtime, interaction-triggered, timed/delayed, broadcast-state, composite/hidden asset, audio-dominant, thumbnail/runtime mismatch, source-only/latent).
- Annotation Confidence: {Low, Medium, High, Confirmed}, based on directness and clarity of observation.
Distinctively, the framework decouples content type from risk and systematically separates evidence location and reveal pathway, enabling fine-grained provenance and actionable classification for downstream curation tasks.
Empirical Audit Design
A stratified sample of 500 Scratch projects was assembled from three pipelines: curated prior examples, taxonomy-keyword guided search, and exploration of high-density clusters identified during review (N ≈ 1,458 initially considered). A hybrid annotation pipeline was employed, combining LLM-assisted runs with human checkpoint adjudication, leveraging a headless scratch-vm harness for automated execution and event simulation. Projects were exhaustively explored for runtime-revealed signals, and exploration depth was systematically coded. The sampling was intentionally biased towards stress-testing the framework rather than estimating platform-wide prevalence.
Key Findings and Quantitative Results
Numerical Highlights
- 93% (467/500) of projects required runtime inspection beyond static metadata to surface safety-relevant signals.
- 77% (387/500) required interaction, gameplay progression, failure states, or deep asset/code examination.
- Only 7% (33/500) could be fully evaluated via static page inspection.
- High category/label prevalence: e.g., 233 flagged for violence/threat, 252 for horror/survival, 99 for political/societal.
- Severity distribution: while 46 (9.2%) were rated as high-risk or likely policy-violating, the majority (248+182=430) fell in the sensitive-but-acceptable or mildly age-inappropriate bands.
Patterns
- Metadata-only curation is insufficient: superficial attributes (title, description, thumbnail) are poor predictors for runtime-exposed sensitive content.
- Reveal pathways are heterogeneous: Content may manifest only after specific interactive inputs or game progression, and a nontrivial proportion used thumbnail/runtime mismatch (M7) to obfuscate risk.
- Risk and category are orthogonal: Category membership (e.g., horror, violence) is not predictive of extremity; stylized combat and explicit gore receive differentiated risk scores.
- Annotation disagreements are systematic: Confusion points (e.g., stylized vs. explicit violence, horror vs. bullying, anti-awareness vs. harm) fall along clear operational boundaries, underlining the necessity for a structured annotation taxonomy.
Theoretical and Practical Implications
The results have direct implications for classroom adoption of public Scratch projects, construction of research datasets, and the development of educator-facing screening tools:
- Mandatory runtime-aware curation: Any screening or selection process that does not explore at least common runtime trajectories will systematically miss safety-relevant content, potentially exposing students to inappropriate material.
- Provenance metadata for datasets: Research datasets should record screening depth, reveal mechanisms checked, and annotation confidence levels. Datasets based purely on static attributes are insufficient for robust secondary analysis or classroom adoption.
- Guidance for automated curation tools: The evidence/reveal taxonomy provides a blueprint for future screening assistants capable of exercising plausible input trajectories, inspecting hidden assets, and flagging runtime/metadata mismatches.
- Nuanced pedagogical policy: Appropriateness is context-dependent—some genres (e.g., mild horror) may fit within elective coursework but not open-ended browsing for younger cohorts. The taxonomy enables calibrated local policy rather than reliance on platform-level content moderation alone.
- Ethical stewardship: The audit emphasizes minimizing harm, ceasing exploration at detection of severe cases, and anonymizing project creators. It also highlights the need for agency-preserving, collaborative content guidance for youth-facing educational platforms.
Limitations and Future Directions
- Sampling bias: The sample is designed to stress-test the annotation framework, not to provide prevalence estimates.
- Exploration completeness: Time and interaction constraints imply that some deeply hidden content may be missed.
- Adjudication reliability: Supervision was conducted by an internal lead; independent double annotation and inter-rater agreement metrics are left for future work.
- Dynamic platform content: Because Scratch projects are mutable (edit, deletion, remix), annotation effort must be temporal and context-aware.
Future developments may include large-scale, automated runtime curation tools leveraging the annotation taxonomy, studies of student/teacher response to runtime evidence during project browsing, and the extension of this framework to other UGC programming platforms.
Conclusion
The study establishes that public Scratch projects exhibit sensitive content that is frequently invisible to static inspection and is often only accessible through runtime interaction, gameplay progression, or hidden asset/code inspection. The introduced runtime-aware annotation framework enables higher-precision, provenance-attached screening aligned with realistic classroom and dataset curation workflows. Adoption of metadata-only curation leaves meaningful safety signals unresolved; therefore, the paper provides a formal vocabulary, numerical evidence, and a policy rationale for runtime-aware educational curation in Scratch and analogous block-based programming communities.
Reference: "Content Hidden Behind Execution: Analyzing Public Scratch Projects at Runtime" (2607.03700)
