Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization

Abstract: Theorem-proving benchmarks evaluate proof search against fixed formal statements, but natural-language-to-Lean formalization must generate the formal statement itself. In this setting, compilation is only a validity check: a Lean declaration may type-check while omitting hypotheses, changing domains, or expressing a vacuous claim. We study faithful statement formalization as both an evaluation problem and a bottleneck-attribution problem. On a 400-entry graduate-level benchmark spanning real analysis, complex analysis, topology, and algebra, our protocol combines Lean compilation, cross-model semantic judging, and human expert calibration. The resulting picture is different from compile-rate evaluation: a full tool-augmented agent reaches 89.5% compilation but only 60.5% consensus faithfulness, exposing a 29.0-point compile-pass but consensus-unfaithful gap. Targeted human audits support the metric as a conservative decision boundary: across available case-level audits, 96.0% of consensus-positive outputs are human-confirmed faithful, while 82.4% of compile-pass consensus-negative outputs are human-confirmed semantic failures. Under this metric, existing one-shot formalizer models and prover-oriented Lean models remain low, suggesting that formal validity, proof-oriented Lean competence, and faithful statement generation should be reported separately. We then use a full $2^3$ factorial design to decompose three recurring interventions in formalization pipelines: parametric expert drafting, Mathlib/context search, and Lean elaboration feedback. Elaboration feedback is the largest validity intervention, but it also exposes a larger compile-pass semantic-failure bucket; search mainly improves grounding and selectivity; and fine-tuned drafting is largely substitutable in this tool stack once feedback and grounding are available.

• The paper establishes that semantic faithfulness in NL-to-Lean formalization requires evaluation beyond mere compilation.
• It employs a rigorous protocol combining compiler checks, strict LLM consensus, and expert calibration to assess fidelity.
• Benchmark results reveal a significant compile-pass gap, highlighting the limitations of proof-oriented training for semantic accuracy.

Faithful Natural-Language-to-Lean Formalization: Evaluation, Bottlenecks, and Tool Effects

Abstract and Problem Context

The paper "Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization" (2606.31002) addresses the problem of translating natural-language mathematics into machine-checkable Lean 4 statements. Unlike the prevailing focus on automated theorem proving, where proof search operates against trusted formal targets, the autoformalization task requires generating the target formal statement itself. In this regime, compilation is an essential check but an insufficient metric for faithfulness: Lean type-checking does not guarantee semantic preservation, with compile-pass outputs potentially omitting hypotheses, shifting domains, or encoding vacuous claims. The authors establish semantic faithfulness as both a core evaluation problem and a bottleneck attribution target, deploying a large-scale graduate-level mathematical benchmark and a protocol combining compiler verification, strict LLM consensus scoring, and human expert calibration.

Evaluation Framework: Semantic Faithfulness Beyond Compilation

The distinction between theorem proving and statement formalization creates a fundamental evaluation difference. In the former, the formal statement is fixed; proof success determines correctness. In formalization, the model must generate the declaration, and mere compilation does not decide semantic alignment. The authors’ evaluation protocol therefore combines Lean compilation as a validity filter with semantic scoring via independent LLM judges (GPT-5.2 and Gemini-2.5-Pro), with strict consensus required for faithfulness.

Human calibration of this protocol demonstrates high positive precision (96.0%), confirming that consensus-positive outputs are nearly always semantically faithful, while most compile-pass consensus-negative outputs are semantic failures (82.4% confirmation). Thus the protocol operates as a conservative, high-precision boundary rather than a recall-oriented equivalence proof.

Benchmark and System Comparison

The benchmark comprises 400 graduate-level statements selected from four mathematical domains: real analysis, complex analysis, topology, and algebra. Source materials are natural-language textbook/lecture notes without pre-existing Lean formalizations, avoiding trivial retrieval or low-level translation.

The comparison spans general-purpose LLMs (e.g., GPT-5.2, Sonnet 4.5, Gemini-2.5-Pro), specialized dedicated formalizers (Herald Translator, Kimina-Autoformalizer, StepFun-Formalizer), prover-oriented Lean models (Kimina-Prover, Goedel-Prover), and tool-augmented agent architectures. Results show that:

• General-purpose LLMs achieve consensus faithfulness rates in the 20–28% range, substantially below compilation.
• Dedicated formalizers and prover-oriented Lean models remain low in faithful statement generation, illustrating that proof-oriented training does not transfer directly to semantic preservation in translation.
• Tool-augmented agents with proper orchestration (all tools enabled) reach 89.5% compilation but only 60.5% consensus faithfulness, revealing a striking 29% compile-pass—but semantically unfaithful—gap.

  Figure 1: Compile rate and semantic faithfulness diverge across model families; tool-augmented agents with full pipeline reach 60.5% consensus faithfulness, exposing the compile-pass gap.

Agent Architecture and Tool Factorial Analysis

The agent architecture comprises three canonical interventions in autoformalization:

• Expert Drafting (T): Fine-tuned translation prior (Herald model).
• Mathlib/Context Search (S): Symbol lookup and retrieval (lean_inspect_name, lean_resolve_name, web search).
• Lean Elaboration Feedback (F): Whole-statement REPL compilation diagnostics.

The orchestrator (GPT-5.2) mediates formal code generation, tool invocation, and repair, exposing non-monotonic tool interactions and enabling trajectory-level analysis.

Figure 2: Agent orchestration logic—central orchestrator coordinates drafting, search, and compilation tools in Lean 4.

A full $2^3$ factorial design evaluates main and interaction effects of these interventions. Key findings:

• Elaboration feedback (F) is the strongest validity intervention, dramatically raising compile rates from ~25% to >88%. However, it also expands the compile-pass/consensus-unfaithful bucket, proving that REPL feedback is not a semantic oracle.
• Search (S) primarily improves semantic grounding and selectivity, with larger uplift when feedback is absent, and increased efficiency when feedback is enabled (REPL calls reduced by >25%).
• Drafting (T) is largely substitutable in the presence of feedback and search; its effect is more leverageable in low-tool regimes.

Domain-Specific Behavior and Judge Disagreement

The domain-level decomposition exposes substantial variation in translation difficulty and tool effectiveness. Complex analysis benefits most from feedback, converting validity gains into faithfulness, while topology and algebra show diminished semantic gains, often due to advanced Mathlib encoding constraints.

Cross-judge disagreement (GPT vs. Gemini) correlates with domain difficulty, with real analysis producing the most conflicts, again aligning with observed bottlenecks.

Figure 3: GPT–Gemini disagreement count by mathematical domain, marking real analysis as most challenging.

Figure 4: Faithful rate by domain and configuration, showing tool-dependent domain effects.

Practical and Theoretical Implications

This work demonstrates that automated statement formalization cannot rely on compilation as an evaluation metric. Semantic robustness mandates calibrated external judgment—currently best operationalized by strict LLM consensus with expert audits. Tool-augmented agents deliver major gains, but the remaining bottleneck is semantic: a large fraction of compile-valid outputs fail to capture the intended mathematical meaning, and this gap is conservative under human review.

The factorial decomposition further shows that no tool is uniformly additive; interventions can substitute, interfere, or simply shift the failure modes, underscoring the need for per-instance routing, early stopping, and semantic-aware repair policies.

For theory, this reveals that formal syntactic acceptance (type-checking, REPL feedback) is decoupled from semantic alignment, emphasizing the necessity for meaning-preservation evaluation and exposing the limits of proof-oriented training for translation-only tasks. For practice, scaling formalization requires strong semantic judging, efficient orchestration, and dynamic tool usage policies—relevant not only for mathematical AI but for broader code synthesis and specification alignment.

Future Directions

Prospects for future development include:

• Integration of richer semantic retrieval systems (e.g., LeanSearch), potentially strengthening search effects and closing domain bottlenecks.
• Automatic early-stopping or routing based on trajectory features, maximizing efficiency and semantic accuracy.
• Pairing tool-augmented formalization with semantic equivalence provers (BEq) or neuro-symbolic metrics for finer-grained evaluation.
• Expansion to broader mathematical domains and research-level statements.
• Exploration of LLM ensemble strategies and judge calibration protocols, mitigating model-family dependence in semantic assessment.

Conclusion

The paper establishes that faithful formalization in Lean demands semantic evaluation in addition to compiler validity. The rigorous, human-calibrated protocol exposes the semantic bottleneck, reshapes system comparison, and motivates design of agents and tools that explicitly distinguish syntax repair from semantic alignment. The practical mandate is clear: future autoformalization systems must combine strong Lean feedback, robust retrieval, and scalable semantic judgment, deploying per-instance diagnostic policies to ensure meaning preservation—not merely successful compilation.