Designing FSMs Specifications from Requirements with GPT 4.0

Abstract: Finite state machines (FSM) are executable formal specifications of reactive systems. These machines are designed based on systems' requirements. The requirements are often recorded in textual documents written in natural languages. FSMs play a crucial role in different phases of the model-driven system engineering (MDE). For example, they serve to automate testing activities. FSM quality is critical: the lower the quality of FSM, the higher the number of faults surviving the testing phase and the higher the risk of failure of the systems in production, which could lead to catastrophic scenarios. Therefore, this paper leverages recent advances in the domain of LLM to propose an LLM-based framework for designing FSMs from requirements. The framework also suggests an expert-centric approach based on FSM mutation and test generation for repairing the FSMs produced by LLMs. This paper also provides an experimental analysis and evaluation of LLM's capacities in performing the tasks presented in the framework and FSM repair via various methods. The paper presents experimental results with simulated data. These results and methods bring a new analysis and vision of LLMs that are useful for further development of machine learning technology and its applications to MDE.

• The paper demonstrates a novel pipeline that leverages GPT-4 to automatically synthesize deterministic FSMs from natural language requirements.
• The methodology integrates iterative prompt engineering with syntactic and semantic error analysis, using fault models and distinguishing sequences for repair.
• The study highlights the practical need for hybrid LLM–formal methods in model-driven engineering to ensure accurate and scalable FSM extraction.

LLM-Driven Synthesis and Repair of FSM Specifications from Natural Language Requirements

Introduction

This work addresses the automated synthesis of deterministic finite state machines (DFSMs), specifically Mealy machines, from natural language requirements utilizing GPT-4.0. FSMs are foundational to model-driven engineering (MDE), enabling formal behavioral specification, systematic test derivation, and aiding verification and validation processes. Manual extraction of FSMs from requirements is resource-intensive and error-prone. The study presents a comprehensive framework leveraging LLMs for generation and rigorous repair of FSMs, contributing significant insights on prompt engineering, systematic error analysis, and exploitation of test-oriented formal methods for enhanced reliability of synthesized models.

LLM-Based FSM Synthesis Pipeline

The proposed methodology involves several steps. The pipeline begins with English descriptions of system behavior, potentially generated via patterns to increase diversity and challenge the LLM. Textual descriptions lack explicit domain cues, demanding contextual inference by the LLM. LLMs are prompted, using task-specific and format-constrained queries, to produce a formal CSV representation of the FSM. The approach is visualized below.

Figure 1: The workflow for using LLMs to synthesize FSMs from textual requirements, including iterative expert-in-the-loop repair strategies.

Prompt engineering is identified as critical, with the experiment showing that even minor variations in prompts can influence the LLM output structure and correctness. Fine-tuning at the prompt level—rather than at model-parameter level—remains integral, given the synthetic data setup where formal retraining is not feasible or necessarily beneficial.

Formal Evaluation of Synthesis Accuracy

Correctness of the generated FSM is grounded in two orthogonal axes: syntactic and semantic equivalence against an oracle FSM.

• Syntactic comparison involves state and transition correspondence. Syntactic faults are classified into:
  • Type 1: Additional transitions (spurious nondeterminism)
  • Type 2: Missing transitions (incompleteness)
  • Type 3: Output faults (incorrect output labeling)
  • Type 4: Transfer faults (incorrect next state assignment)
• Semantic comparison utilizes distinguishing sequences—input strings that elicit observable output differences between the generated FSM and the oracle. This is formalized via construction of a distinguishing automaton, which enables identification of minimal separating sequences in linear time, provided the machines are input-complete.

Empirical results over FSMs of 5, 10, and 25 states—with 30 random instances per configuration—demonstrate that GPT-4.0 achieves high accuracy on small machines (mean faults per machine < 0.1 for 5 states) but error rates and complexity of faults (maximum up to 11) escalate with increasing size/complexity. Output faults and missing transition faults are predominant. This result has practical implications for scaling LLM-assisted formal model synthesis.

Iterative Repair Methodologies

Recognizing that generated FSMs frequently require correction, the study operationalizes repair as iterative prompt refinement, informed by diagnostic feedback grounded in model-based testing theory. Four primary approaches are evaluated:

Syntactic Fault-Based Repair

When an FSM oracle is available, syntactic faults are explicitly enumerated. Prompts are augmented with synthetic description fragments instructing the LLM to add, remove, or conserve specific transitions. This method achieves a 100% repair rate for all tested configurations, suggesting LLMs can effectively integrate explicit structural corrections.

Distinguishing Sequence-Based Repair

Faults are conveyed as input-output traces that differentiate the generated FSM from the oracle. Prompts include sequences that must yield specified output behaviors. While this method retains theoretical appeal (being more abstract and less reliant on oracle structure), the empirical repair rate drops sharply as machine size increases (0% for 10 states), likely a result of LLM context limitations and degraded attention with long, less structured prompt augmentations.

Checking Sequence-Based Repair

In scenarios mimicking practical process, experts can only validate outputs on checking sequences for generated machines. Checking sequences, though uniquely identifying for a specified number of FSM states, are computationally hard to derive. Results show a lower repair rate and practical constraints, notably due to computation overhead and potential for repeated synthesis failures.

Fault Model-Based Repair with Mutation Machines

This method leverages a priori knowledge of LLM-induced fault profiles to define a repair domain (a mutation machine) encompassing likely correctable candidates. An iterative process, query-efficient and guided by distinguishing sequences within the domain, systematically eliminates non-conforming FSMs via expert output validation. This approach achieves 100% repair rate in all tested cases, with the caveat of infrequent necessity to further augment the mutation machine, highlighting the value of explicit LLM fault modeling.

Numerical Observations

• For FSMs with 5 states, GPT-4.0 could successfully synthesize correct specifications from natural language in 93.3% of cases (28/30).
• Syntactic fault-based repair always converged in 1–3 iterations.
• Semantic repair using distinguishing sequences failed to resolve faults for more complex FSMs.
• Mutation machine-guided repair led to 100% recovery, with maximal expert queries required being 30 (5 states) and 89 (10 states).

Implications and Future Directions

This investigation elucidates both the capabilities and limitations of state-of-the-art LLMs in formal synthesis tasks. While LLMs can infer small to moderately complex FSMs from informal language, they are not robust to scaling and are sensitive to both prompt structure and fault propagation. The findings reinforce the necessity of hybrid LLM–formal method workflows, augmenting generative ability with systematic test-oriented repair and expert validation.

Practical Implications:

• LLMs can accelerate initial FSM modeling from requirements, but formal model checking and automated repair remain essential for critical domain applications.
• Semi-automatic repair workflows merit further research, especially for reducing the number of expert queries and prompt convergence bottlenecks.
• Integration of domain-specific language cues and inductive bias in prompts remains a promising avenue to enhance LLM performance.

Theoretical Implications:

• The repair process effectively constitutes an active learning paradigm, leveraging oracles, distinguishing sequences, and expert knowledge in a structured framework.
• Mutation machines serve as generative fault domains, enabling both coverage analysis and informed, constrained correction.

Speculation on Future Developments:

Advances in LLM context management, prompt interpreters, and integration with formal analysis/verification backends could extend this approach to large-scale, real-world MDE scenarios. Incorporation of symbolic automata representations and applications to heterogeneous specification languages (e.g., UML, Verilog) remain open challenges.

Conclusion

This study presents a systematic evaluation of LLM-based FSM synthesis and multifaceted repair strategies, highlighting both the strengths and limitations of LLMs for formal specification tasks. By integrating prompt engineering, syntactic/semantic evaluation, and model-based repair methods, the approach offers a pathway toward robust FSM extraction in MDE pipelines, with direct applicability to software testing, system verification, and automation engineering. The findings suggest that while LLMs offer valuable automation for FSM synthesis from requirements, their outputs must be systematically validated and repaired, leveraging the full toolbox of formal methods and interactive expert-in-the-loop techniques.

Reference: "Designing FSMs Specifications from Requirements with GPT 4.0" (2603.29140)