IsabeLLM: Automated Theorem Proving Applied to Formally Verifying Consensus

Abstract: Advances in AI have led AI for Theorem Proving to become a promising means of formally verifying computer systems. Whilst formal verification is traditionally reserved for safety-critical systems due to the required amount of expertise and effort, AI can help to automate a large amount of this workload and make it far more accessible. Blockchain-based systems are becoming increasingly popular and are frequently targeted by malicious actors, often resulting in huge financial losses, highlighting the need to better verify these systems and mitigate vulnerabilities. Arguably the most important component of these systems is the consensus protocol, which allows nodes to agree on decisions in a potentially adversarial environment. In this paper, we improve upon IsabeLLM, the automated theorem proving tool in Isabelle. Namely, we implement a Retrieval-Augmented Generation framework, Error tracing and counterexample generation for improved context supplied to the LLM. Compatibility with the latest version of Isabelle and Sledgehammer is also implemented for improved efficiency. We compare the performance of the two versions of IsabeLLM in their ability to complete the verification of Bitcoin's Proof of Work consensus.

• The paper introduces IsabeLLM-RAG, an innovative framework that enhances formal verification of Bitcoin's Proof-of-Work consensus via retrieval-augmented LLM techniques.
• The methodology integrates error tracing, counterexample generation with Nitpick, and iterative feedback loops to improve proof search efficiency and success rates.
• Key experimental results show significant improvements over prior work, achieving up to a 94.4% success rate and reducing average proof attempts per lemma.

Formal Verification of Blockchain Consensus with Automated Theorem Proving: An Analysis of IsabeLLM

Introduction

This work addresses the challenge of formal verification in the context of blockchain consensus protocols, specifically extending automated theorem proving to verify the correctness of Bitcoin's Proof-of-Work (PoW) consensus mechanism. Formal verification is a critical tool in blockchain, given the irreversible consequences of deployment errors and the financial impact of protocol vulnerabilities. Traditional formal verification has been underutilized due to its technical complexity and resource demands. The integration of Artificial Intelligence—particularly LLMs—into proof assistants offers the prospect of democratizing the verification process and expanding its applicability to complex, real-world protocols.

In this context, the paper proposes IsabeLLM-RAG, an enhanced automated theorem proving framework for Isabelle, built upon prior work with IsabeLLM. Key novelties include a retrieval-augmented generation (RAG) database, integration of counterexample generation with Nitpick, error tracing for iterative LLM refinement, improved compatibility with Isabelle 2025 and its tools, and a comparative evaluation across modern LLM architectures.

Blockchain Consensus and Formal Modeling

The verification target is the Bitcoin PoW consensus protocol. The model generalizes from binary trees to n-ary trees, significantly increasing the structural complexity and the corresponding logical proof space. The primary property of interest is the common prefix property, essential for safety in distributed consensus, as formalized in the Bitcoin Backbone model. The formalism involves inductive definitions capturing honest and adversarial event traces and the recursive criteria for blockchain growth, parameterized by honest and dishonest transitions. This generalization is motivated to create a more realistic blockchain abstraction and to rigorously test the scalability of automated theorem proving in this domain.

IsabeLLM Architecture and Workflow

IsabeLLM provides an interactive layer integrating the Isabelle proof assistant, Sledgehammer automation, and an LLM invocation and feedback loop. Its workflow consists of iterative cycles: extracting the next unproven lemma, generating candidate proofs with the LLM, integrating these into the Isabelle theory file, and applying Sledgehammer-based automation for failed steps. Any errors encountered in build or tactic execution trigger a feedback loop to the LLM, now augmented with an error trace and RAG context.

Figure 1: IsabeLLM Architecture linking Isabelle, Sledgehammer, and the LLM for iterative theorem proving.

The system's architecture centers around resilient error handling: detection of incomplete (sorry) proofs, failed tactic invocations, and hallucinated facts or syntax are all systematically passed to the LLM for correction. IsabeLLM is implemented predominantly in Scala for tight coupling with Scala-Isabelle, and leverages Python for LLM API interactions.

Figure 2: An example IsabeLLM workflow showcasing generation, correction, and integration of the subtree_height lemma.

Retrieval-Augmented Generation, Error Tracing, and Counterexample Generation

A principal innovation is the integration of RAG. A static database of prior related proofs is indexed and vectorized with sentence transformers. For each proof obligation, the k-nearest neighbors (by embedding distance) are retrieved and attached to the LLM prompt, offering domain-specific, structurally similar context. This serves both as an inductive bias and as guidance for proof structure in domains with sparse or technical corpora, such as formal consensus verification.

Error tracing was introduced to improve sample efficiency: the full sequence of edits and corrective steps between LLM calls is appended to subsequent prompts, preventing regression and repeated modification of steps already validated by the proof assistant or Sledgehammer.

Counterexample support via Nitpick is embedded into the loop. Incorrect conjectures or faulty intermediate steps are identified automatically, and the counterexample is supplied to the LLM, making explicit the semantic landscape that must be navigated and accelerating convergence towards a correct proof.

Figure 3: Example counterexample generated by Nitpick, enabling rapid identification and correction of logically false proof steps.

Model Comparison and Experimental Results

The evaluation benchmarks IsabeLLM-RAG with leading LLMs: DeepSeek R1T2 Chimera, NVIDIA Nemotron 3 Super, and OpenAI GPT-OSS-120B, comparing against the original DeepSeek R1 baseline from prior work. The metric is success rate across 16 non-trivial lemmas for the n-ary consensus model, with per-lemma iterative success and efficiency (average iterations per lemma).

Substantial improvements are claimed:

• Chimera achieved a 94.4% success rate (vs. 76.3% for DeepSeek R1 baseline), with the lowest average proof attempts per lemma (1.06 vs. 1.31).
• Nemotron outperformed its parameter-efficient design expectations with 87.5% success, suggesting MTP innovation compensates for smaller active parameter count.
• GPT-OSS performance is more limited (67.5%), likely due to aggressive quantization and a low number of active parameters.

Most notably, complex lemmas requiring deep inductive reasoning (e.g., branch_height, foldr_max_eq) saw a marked increase in successful automation, where prior work failed or required extensive manual Sledgehammer intervention. The proof search space becomes more deterministic and less prone to model hallucination due to RAG anchoring and context pruning. Additionally, the main failure modes (e.g., proof loops, repeated syntactic errors) were nearly eliminated with the new error tracing and counterexample mechanisms.

Limitations and Implications

Despite the advances, IsabeLLM remains constrained to proof completion, not conjecture generation. Complete automation across the full specification-discovery-verification pipeline is not realized; users must still supply system models and invariants. The integration with frameworks like Isabelle/Solidity can mitigate this by reducing user burden for protocol and contract modeling, but the challenge of deep domain adaptation to novel calculi remains, given the paucity of machine-interpretable formal corpora in specialized areas.

Scaling to larger verifications is limited by context window constraints. While RAG provides effective static retrieval, for large codebases, dynamic retrieval approaches such as the Model Context Protocol (MCP) are necessary to support proof search over unbounded corpora.

Pragmatically, IsabeLLM enables a step change in accessibility for non-trivial formal verifications of consensus protocols, smart contracts, and more broadly, safety-critical systems. The approach directly supports discovery of proof artifacts, potential new proof strategies, and quicker identification of edge-case failures via integrated counterexample analysis.

Future Directions

Potential enhancements include LLM parallelization for distributed proof search, application-specific fine-tuning of models on formal proof corpora, and seamless integration with domain-specific formal languages. Furthermore, improvements in semantic code embedding and dynamic retrieval will be essential for verification tasks beyond single theory files. As LLM capabilities mature, full automation of the specification phase—conjecture mining and candidate invariant synthesis—could be realized, closing the loop for end-to-end formal synthesis and verification.

Conclusion

IsabeLLM-RAG advances the state-of-the-art in automated theorem proving applied to protocol verification, delivering significant improvements in success rates, proof search efficiency, and resilience to semantic and syntactic failure. These advances are achieved through a principled combination of retrieval-augmented LLM prompting, error-aware prompt engineering, and counterexample-guided refinement. The framework demonstrates scalability to more complex models and highlights key constraints and future research directions for full AI-powered formal verification in distributed and adversarial systems (2606.18098).