Large Language Models Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles

Abstract: Recent work has explored the use of LLMs for generating tutoring responses in mathematics, yet it remains unclear how closely their instructional behavior aligns with expert human practice. We examine this question using a controlled, turn-level comparison in which expert human tutors, novice human tutors, and multiple LLMs respond to the same set of math remediation conversation turns. We examine both instructional strategies and linguistic characteristics of tutoring responses, including restating and revoicing, pressing for accuracy, lexical diversity, readability, politeness, and agency. We find that LLMs approach expert levels of perceived pedagogical quality on average but exhibit systematic differences in their instructional and linguistic profiles. In particular, LLMs tend to underuse restating and revoicing strategies characteristic of expert human tutors, while producing longer, more lexically diverse, and more polite responses. Statistical analyses show that restating and revoicing, lexical diversity, and pressing for accuracy are positively associated with perceived pedagogical quality, whereas higher levels of agentic and polite language are negatively associated. Overall, recent LLMs exhibit levels of perceived pedagogical quality comparable to expert human tutors, while relying on different instructional and linguistic strategies. These findings underscore the value of analyzing instructional strategies and linguistic characteristics when evaluating tutoring responses across human tutors and intelligent tutoring systems.

• The paper demonstrates that LLMs can achieve expert-level pedagogical quality in math tutoring while adopting unique instructional strategies.
• The analysis uses a multi-source dataset and transformer-based classifiers to measure key instructional moves such as pressing for accuracy and revoicing.
• The paper finds that higher lexical diversity boosts pedagogical quality, whereas increased politeness and agency can negatively impact instructional effectiveness.

Pedagogical Quality of LLMs in Mathematical Tutoring: Instructional and Linguistic Divergence

Introduction

The paper "LLMs Approach Expert Pedagogical Quality in Math Tutoring but Differ in Instructional and Linguistic Profiles" (2512.20780) provides a systematic analysis of the capacity of LLMs to deliver pedagogically effective mathematical tutoring. The study contrasts expert human tutors, novice human tutors, and several state-of-the-art LLMs in a controlled setting, examining their responses to the same set of mathematical remediation dialogues. The evaluation framework incorporates both instructional and linguistic measures, facilitating nuanced characterization of response quality and style.

Methodology

A multi-source dataset comprising 2476 tutoring responses to middle and elementary school math errors forms the analytic foundation. Each student mistake triggers responses from two human tutors (expert and novice) and up to seven LLMs (GPT-4, Gemini, Sonnet, Mistral 7B, Llama-3.1-405B, Llama-3.1-8B, Phi-3). Responses are annotated along four core pedagogical axes: mistake identification, mistake localization, guidance provision, and actionability. A composite pedagogical quality score is computed for comparative and inferential analyses.

Instructional moves are quantified using transformer-based classifiers, outputting continuous scores for "pressing for accuracy" and "restating/revoicing." Linguistic features encompass response length (log tokens), lexical diversity (MTLD), readability (Flesch–Kincaid), politeness (pragmatic classifier), and agency (agentic classifier). Regression models and within-turn normalization techniques are used to isolate tutor-specific effects and control for conversational context.

Comparative Instructional and Linguistic Profiles

Analysis shows robust and systematic differences between LLM and human tutoring profiles:

• Instructional Moves: LLMs surpass experts in "pressing for accuracy" but are consistently deficient in "restating/revoicing," a hallmark of expert feedback. Novice tutors similarly underutilize restating/revoicing, suggesting its link to instructional expertise.
• Response Length and Lexical Diversity: LLM outputs are substantially longer and more lexically diverse than expert and novice tutors.
• Readability: All LLMs yield lower Flesch–Kincaid scores (less readable) than expert humans.
• Politeness and Agency: LLMs display higher politeness and variable agency compared to experts. Novices exhibit higher agency but moderate politeness.

Pedagogical Quality Across Tutor Types

LLMs can attain average pedagogical quality scores comparable to expert tutors, contingent on model architecture. Novice tutors consistently underperform relative to both experts and higher-performing LLMs within the same conversational contexts. This parity is notable given that LLMs rarely mimic the instructional and linguistic configurations of experts.

Feature–Outcome Associations

Regression analysis provides high-resolution insights into correlates of pedagogical quality:

• Strong positive associations: Restating/revoicing ($\beta = 0.78$, $p < 0.001$), lexical diversity ($\beta = 0.62$, $p < 0.001$), and pressing for accuracy ($\beta = 0.17$, $p = 0.007$).
• Negative associations: Politeness ($\beta = -0.19$, $p < 0.001$) and agency ($\beta = -0.68$, $p < 0.001$).
• Null effects: Readability ($\beta = 0.08$, $p = 0.151$) and response length ($\beta = 0.19$, $p = 0.059$) do not significantly predict pedagogical quality.

A bold finding is that increased politeness and agentic language correlate negatively with perceived pedagogical quality, contradicting informal intuitions that such traits would enhance instructional desirability. Instead, the findings emphasize the importance of targeted revoicing and vocabulary richness.

Theoretical and Practical Implications

The study demonstrates that pedagogical effectiveness in both human and LLM tutors predominantly resides in specific instructional behaviors and linguistic configurations, not in tutor source or superficial fluency. LLMs, when evaluated in turn-centered contexts, often approximate expert human performance but arrive there through distinct strategic and linguistic pathways. The dissociation between politeness/agency and pedagogical quality signals potential pitfalls for human-centered LLM fine-tuning regimes that prioritize affability or forceful guidance at the expense of instructional specificity.

Theoretically, these results support models of instructional efficacy that foreground turn-level microstructures—including revoicing and error-focused scaffolding—over stylistic or macro-discursive features. Practically, the findings challenge both LLM design practices and educational deployment: optimizing for human mimicry or maximizing politeness is unlikely to yield further improvements in pedagogical impact. Instead, aligning LLMs to expert-validated instructional moves, especially revoicing, appears critical.

Future Developments

Anticipated advances in AI tutoring depend on integrating turn-level, feature-informed reward signals into LLM fine-tuning frameworks. Expanding feature sets to include discourse strategies, epistemic framing, and domain-specific problem decomposition may further sharpen instructional alignment. Evaluations coupling response-level pedagogical scores with behavioral or learning outcomes in multi-turn interactions will be essential for validating the generalizability and instructional utility of LLM tutors across subjects and languages.

Conclusion

This paper provides a rigorous framework and empirical baseline for mathematical tutoring evaluation by LLMs. The main conclusion is that LLMs can reliably reach expert-level pedagogical quality, but do so through divergent instructional strategies and linguistic forms. Effective feedback in this context is not a function of length, fluency, or politeness, but rather of targeted instructional moves—especially revoicing and diversity of expression—aligned with error diagnosis and guidance. For AI tutoring development, this mandates a transition from stylistic mimicry to the emulation of pedagogically validated response features.