Evaluating Interactivity: Toward Automated Assessment of AI-Generated Explorable Explanations

Abstract: While LLMs now enable rapid generation of interactive learning materials, evaluating the interaction quality of these explorable explanations remains an open challenge. Existing benchmarks largely focus on code executability or visual fidelity, providing limited insight into dynamic interaction behaviors such as learner-controlled state transitions and context-sensitive system responses, which are factors that critically shape learners' conceptual understanding. We present EE-Eval, an automated evaluation framework that formalizes interactivity as a finite space of learner-controllable states and transitions, represented as a Finite State Machine (FSM). By extracting FSMs from AI-generated explorable explanations, EE-Eval externalizes implicit interaction logic into an explicit, machine-interpretable graph. Evaluation is performed by comparing each generated FSM to an ideal FSM that encodes pedagogical intent, using a combination of graph-based metrics and embedding-based comparison of states, actions, and feedback to measure their structural and semantic similarity. Across thousands of generated explorable explanations spanning 127 concepts and produced by 6 AI models, EE-Eval consistently differentiates interaction quality beyond surface-level criteria such as functional correctness or visual quality, and exhibits substantially stronger alignment with human judgments of interactivity and pedagogical effectiveness than existing baselines. By framing interactivity as testable behavioral models rather than an emergent byproduct of LLM generation, EE-Eval transforms evaluation into a reflective diagnostic tool, enabling pedagogically grounded and actionable human-AI collaboration in creating interactive educational content.

• The paper introduces EE-Eval, an FSM-based evaluation framework that extracts and compares interactivity in AI-generated explorable explanations.
• The paper demonstrates its framework's superior alignment with human ratings by using structural, semantic, and behavioral metrics achieving significant correlation values.
• The paper highlights how measuring interactive behavior provides actionable diagnostics to refine AI-generated educational content and improve pedagogical outcomes.

FSM-Based Evaluation of Interactivity in AI-Generated Explorable Explanations

Motivation and Background

AI-driven explorable explanations are increasingly used to facilitate active learning in domains requiring computational reasoning, yet their evaluation has historically been restricted to static visual outputs or isolated functional correctness, omitting the critical facet of interactive behavior. Existing benchmarks, such as nvBench and VisEval, assess visual fidelity and code executability but provide minimal insight into the dynamic learner-system interaction that underpins conceptual understanding. The paper "Evaluating Interactivity: Toward Automated Assessment of AI-Generated Explorable Explanations" (2606.31012) addresses this gap by formalizing interactivity through explicit behavioral modeling with Finite State Machines (FSMs) as a standard abstraction for analysis and benchmarking.

EE-Eval Framework: Formalizing Interactivity via FSMs

EE-Eval externalizes the implicit interaction logic embedded within AI-generated HTML/JavaScript artifacts by representing explorable explanations as FSMs. Each FSM describes:

• States: Semantically meaningful learner-controllable configurations
• Transitions: User-initiated or conditional state changes
• Components: UI elements bound to interaction pathways
• Metadata: Pedagogical intent, context, and variable definitions

Extraction of FSMs utilizes a few-shot LLM-based approach, leveraging manually validated HTML–FSM pairs for robust mapping and schema conformity. This explicit FSM representation enables normalization across models and facilitates direct comparison of interaction structures independent of implementation details.

Figure 1: The explorable explanation interface and corresponding extracted FSM graph, illustrating state-transition logic externalized from complex UI code.

Evaluation Metrics and Automated Analysis Pipeline

EE-Eval operationalizes its evaluation by constructing ideal FSMs encoding pedagogically motivated interaction, followed by multi-dimensional similarity analysis between generated and reference FSMs:

• Structural Similarity: Quantifies graph topology alignment via node/edge counts, degree distributions, and density.
• Semantic Similarity: Employs MiniLM-based sentence embeddings to compare state labels, actions, events, and metadata, ensuring robustness to lexical variation.
• Behavioral Coherence: Computes FSM isomorphism scores, leveraging canonical node degree and semantic categories to test exact match or behavioral equivalence.

A weighted aggregate score ($S_{\text{total}}$) integrates these dimensions using optimized triplet weights $(\alpha = 0.4, \beta = 0.4, \gamma = 0.2)$, established through grid search on validation data for maximal alignment with human ratings.

Figure 2: The EE-Eval automated multi-agent pipeline illustrating FSM extraction, similarity evaluation, and reporting architecture.

The pipeline efficiently processes explorable explanations generated by multiple LLMs, providing both batch analytics and informative diagnostics for model and artifact refinement.

Figure 3: EE-Eval's interfaces for FSM-based evaluation results and human-rated interactive effectiveness.

Experimental Results: Validity, Alignment, and Discriminative Power

Human Alignment

Evaluation of 2,497 explanations spanning 127 computer science concepts and six major LLMs demonstrated that FSM-based metrics yield the highest overall and interaction-specific correlation with human ratings. In balanced datasets, FSM-based scores achieved Pearson correlations up to $r=0.728$ for interactivity (p<0.001), outperforming both VLM-based visual surrogates and DOM-level unit tests, which performed with weaker or negative alignment, especially in the presence of complex UI designs.

Robustness to Visual Bias

On visually-biased datasets, FSM-based evaluation retained relative sensitivity to interactive quality ($r=0.210$), while vision-oriented baselines correlated with visual style but failed to capture behavioral coherence. This highlights the value of FSM abstraction for mitigating superficial assessment, supporting interactive pedagogical analysis even when appearance cues dominate.

Discriminative Capacity

EE-Eval’s FSM-based similarity distribution distinguished model capabilities with strong statistical support. For example, GPT-5 Mini outperformed GPT-3.5 Turbo in interaction quality ($\Delta\mu = 8.54$, $p<0.0001$), but not in coding-oriented leaderboards, revealing that static benchmarks do not reliably capture interaction logic. The framework exposes that higher-capacity models may produce visually correct yet behaviorally fragmented interfaces, whereas mid-scale LLMs often externalize coherent state-transition structures.

Figure 4: FSM-based similarity score distributions by model, revealing model-specific variations in interaction logic fidelity.

Implications, Limitations, and Future Directions

EE-Eval reframes evaluation from merely visual or functional compliance to rigorous behavioral assessment, supporting interpretable and actionable diagnostics for both model development and educational content authorship. The explicit FSM layer allows educators and researchers to interrogate interactive affordances, coherence, and coverage, rather than relying solely on static manifestations.

Practically, EE-Eval can be embedded within iterative generation pipelines to provide structured feedback for LLM fine-tuning on interactive quality, closing the loop between human intent and model output. Theoretically, it introduces a scalable benchmark for comparative human–AI collaboration, emphasizing interaction as a critical but previously under-measured dimension of AI-supported pedagogy.

Several caveats are noted:

• FSM abstraction may oversimplify continuous or highly emergent interactions.
• Extraction quality is contingent on prompting and normalization robustness.
• Reference FSMs encode domain assumptions that may not generalize across curricula.
• Current focus is on interaction structure; content correctness and learning efficacy remain separate dimensions.

Integration of FSM-based behavioral metrics with vision-based and content-oriented signals, and the extension to probabilistic or hybrid state models, are promising research directions. EE-Eval’s interpretable framework can drive evaluation-informed generation pipelines, ultimately supporting more transparent, reliable, and learner-centered AI-enhanced educational systems.

Conclusion

EE-Eval advances automated evaluation for AI-generated explorable explanations by formalizing interactivity as explicit FSMs, rigorously capturing state-transition logic and behavioral coherence. Experimental evidence demonstrates superior alignment with human judgments and robust discrimination of model and content quality, positioning EE-Eval as a foundational analytic and pedagogic tool for the next generation of interactive educational AI systems.