On the Limitations of Large Language Models for Conceptual Database Modeling

Abstract: This article analyzes the use of LLMs as support for the conceptual modeling of relational databases through the automatic generation of Entity-Relationship (ER) diagrams from natural language requirements. The approach combines different LLMs with prompt engineering techniques to evaluate their ability to identify entities, relationships, and attributes in a conceptually consistent manner. The experimental evaluation involved three LLMs, each subjected to three prompting techniques (Zero-Shot, Chain of Thought, and Chain of Thought + Verifier), applied to the same requirements scenario with progressively increasing complexity. The generated diagrams were qualitatively analyzed through direct comparison with the textual requirements, considering the structural and semantic adherence of the modeled elements. The results indicate that, although LLMs show reasonable performance in less complex scenarios, their reliability decreases as the complexity of the requirements increases, with a rise in inconsistencies, ambiguities, and failures in representing constraints. These findings reinforce that, in their current state, LLMs are not sufficiently mature for reliable use in complex scenarios, and the cost of validation may offset the apparent productivity gains.

• The paper demonstrates LLMs’ limitations in producing robust ER diagrams due to overspecification and underspecification biases.
• It employs a systematic evaluation of models using various prompting strategies, including Chain-of-Thought and Verifier methods.
• Results indicate that both models lack domain-invariance and require expert oversight to achieve reliable conceptual database design.

Limitations of LLMs in Conceptual Database Modeling

Introduction

This paper conducts a rigorous evaluation of LLMs as automated agents for conceptual database modeling, specifically for transforming natural language requirements into Entity-Relationship (ER) diagrams. By systematically analyzing the performance of advanced LLMs (Qwen-3 and GPT-5.1) and the impact of various prompt engineering strategies (Zero-Shot, Chain-of-Thought, Chain-of-Thought+Verifier), the study elucidates the current capabilities and critical limitations of LLMs in supporting high-integrity conceptual abstraction, an essential task in robust database engineering.

Background and Related Work

Classical NLP-driven ER diagram generation frameworks rely on extensive language-specific heuristics, resulting in brittle and low-portability systems that struggle with ambiguity, domain transfer, and integration of complex modeling constructs. Conversely, LLM-based approaches promise language-agnostic abstraction by leveraging foundation models trained on vast, diverse corpora. Prior efforts, however, have not systematically isolated ER diagram generation as a unique challenge nor subjected models to controlled scenario-based evaluations that stress their capacity for semantic completeness, structural correctness, and abstraction fidelity.

The literature review distinguishes the present approach by three factors:

• Exclusive focus on ER generation as opposed to mixed diagramming or broad code synthesis [diagrammerGPT2023].
• Construction of a task-specific benchmark with progressive scenario complexity.
• Methodical application of prompt engineering techniques known to enhance LLM cognitive structuring [wei2022cot, bansal2024prompt].

Experimental Design and Methodology

The evaluation leverages a controlled scenario—hospital access management—parametrized at escalating levels of requirements complexity. Critically, requirements are articulated in domain-level language devoid of modeling jargon to simulate realistic client specifications. For each scenario, ground-truth ER diagrams are constructed as gold standards.

The core framework (Figure 1) automates requirements transformation, model interaction, structural postprocessing, and diagram rendering, supporting reproducibility and precise isolation of failure modalities.

Figure 1: The multi-stage pipeline, from LLM-driven requirement inference to diagram normalization and automated ER rendering.

Model selection was guided by state-of-the-art reasoning benchmarks:

• Qwen-3: An open-weights model optimized for instruction fidelity and structured output.
• GPT-5.1: A proprietary model offering superior adaptive and logical reasoning.

Prompting regimes include One-Shot, Chain-of-Thought (CoT), and CoT+Verifier, enabling assessment of both generation and critical self-assessment capacities.

Evaluation adheres to a qualitative ordinal ranking (L1-L4), holistically considering completeness, correctness, clarity, simplicity, flexibility, and implementability. The stringent schema enforces high standards for conceptual abstraction, rather than superficial or surface-level adequacy.

Results: Model and Prompting Failures

GPT-4.1 Performance

GPT-4.1 (referenced in the experimental section, likely intended as GPT-5.1, maintaining original naming for precision) exhibits a pronounced bias toward over-specification:

• Introduction of superfluous entities (e.g., generalized "PersonRole") and unrequested relationships.
• Omits adherence to fundamental principles of redundancy minimization and functional dependency preservation.
• Fails to distinguish between essential and derived attributes, resulting in structurally unstable and semantically ambiguous diagrams.

Critically, such "creative" additions may align with plausible domain extensions but do not respect the immutability of the requirements. This undermines both the integrity and maintainability of the resulting schema.

Qwen Model Performance

Qwen demonstrates the antithetical bias, characterized by aggressive underspecification:

• Omission of key entities and relationships integral to the domain ontologies.
• Semantic collapse of distinct roles (e.g., physicians, residents, nurses) into a single "Employee" entity, nullifying specific attribute constraints.
• Misinterpretation of relationship cardinalities and temporality, yielding incomplete and non-traceable models, especially for task-critical constructs such as access log traceability.

Both LLMs, therefore, fail to internalize domain invariants and consistently map requirements to structurally and semantically faithful conceptual schemas.

Comparative Prompting Analysis

Employing the Chain-of-Thought (CoT) strategy yields improvement over naive (baseline) prompting by eliciting explicit intermediate reasoning and structural deliberation. Nevertheless, the Verifier augmentation fails to systematically identify and rectify high-level conceptual faults, revealing that superficial self-critique within current LLMs is insufficient for tasks requiring rigorous consistency and multidimensional abstraction.

Theoretical and Practical Implications

The central empirical finding is that neither open nor proprietary, nor baseline nor advanced prompt-engineered LLMs, are currently capable of producing conceptual ER diagrams that satisfy critical thresholds of completeness, correctness, and robustness necessary for professional database design.

Strong claim: None of the evaluated models achieve even Level 3 (structural robustness and data integrity) under the strict qualitative schema; notably, the open-weight Qwen model often fails to satisfy even the most basic L1 requirements.

Consequently, while LLMs may provide plausible outputs in low-complexity tasks, reliance on their output for high-stakes modeling is not advisable—at best, they serve as draft generators requiring expert review; at worst, they introduce systemic biases and defects with significant downstream impact on data quality and application logic.

Future Directions and Open Challenges

Proposed future work focuses on repositioning LLMs in the conceptual modeling pipeline:

• Instead of full automation, use LLMs as partial assistants—e.g., surfacing ambiguities in requirements, acting as semantic validators, or proposing candidate entity sets for human-in-the-loop refinement.
• Develop hybrid evaluation protocols that combine qualitative and quantitative indicators, enabling more reproducible and less subjective assessments of ER model adequacy.
• Investigate fine-tuning LLMs with explicit modeling knowledge and schema validation criteria to bridge observed abstraction and reasoning gaps.

The challenge remains to endow LLMs with domain-invariance recognition, fine-grained constraint modeling, and the ability to distinguish between mandatory and derived structure in the absence of explicit domain cues.

Conclusion

Despite recent advances in large-scale pretraining and sophisticated prompt engineering, current LLMs do not meet professional standards for automated conceptual database modeling from natural language requirements. The models' tendencies toward overspecification or underspecification, inability to respect domain invariants, and lack of principled structural reasoning preclude their unsupervised use in mission-critical systems.

The continued evolution of LLM-driven conceptual modeling will require research into domain-aware prompting, human-in-the-loop workflows, and integrated validation regimes. Until such advances materialize, the expertise and critical judgment of human database architects remain essential in safeguarding semantic integrity and structural soundness in conceptual database design.