Exploring the Semantic Gap in Agentic Data Systems: A Formative Study of Operationalization Failures in Analytical Workflows

Abstract: LLMs are increasingly used to generate queries, invoke tools, and construct analytical workflows. Although recent advances have substantially improved workflow generation and execution, the semantic information required to operationalize analytical concepts often lies beyond what is explicitly represented in database schemas and data values. We present a cross-domain formative study of operationalization failures in agent-generated analytical workflows. Across 236 analytical intents spanning finance, human resources, and public safety domains, we identify 153 recurring failures despite successful workflow generation and execution. Our analysis reveals five recurring classes of failures: comparative grounding, process reasoning, quantitative reasoning, role confusion, and policy grounding. These findings suggest a semantic gap between user-level analytical concepts and the information available to workflow-generation systems. More broadly, they raise questions about the admissibility of analytical operations and suggest that future agentic data systems may require richer semantic representations to bridge the gap between analytical intent and executable computation.

• The paper demonstrates that 65% of workflows fail to operationalize user intents correctly due to a semantic gap between conceptual analytics and available data.
• The paper employs a formative study on 236 natural language intents across finance, public safety, and HR, uncovering recurring failure classes like Comparative Grounding and Role Confusion.
• The paper proposes integrating external semantic artifacts or intermediary representations to better align computational processes with user-level analytical intent.

The Semantic Gap in Agentic Data Systems: Formative Analysis of Operationalization Failures

Introduction

Recent advancements have established LLMs as critical primitives in AI-native data systems for automated query generation, tool invocation, and analytical workflow construction. Notable progress has been achieved in user intent interpretation, schema linking, and producing executable workflows in NL2SQL and agentic data systems. However, a critical limitation persists: the inability to accurately operationalize user-level analytical concepts using only the information typically available in database schemas and raw data. This study systematically interrogates this "semantic gap" by analyzing operationalization failures in agent-generated analytical workflows, across diverse domains including finance, human resources, and public safety.

Figure 1: Illustration of the semantic gap in agentic data systems. Information required to operationalize analytical concepts may not be fully available to workflow-generation systems, leading to operationalization failures despite successful execution.

Problem Formulation: The Analytical Semantic Gap

While prior research has heavily focused on improvements in semantic parsing, schema linking, and reasoning over structured data, less attention has been paid to failures arising after a workflow is both generated and executed successfully. This work isolates the problem where a workflow, despite technical correctness, fails to provide an answer that semantically aligns with the user’s analytical intent. The authors identify this as an instantiation of a semantic gap—the disjunction between user-level conceptual notions (e.g., "unusual risk," "most demanding jobs") and the schematic and statistical context typically available to contemporary agentic workflow generation systems.

Formative Study Design and Methodology

The study investigates 236 natural language analytical intents distributed across real-world datasets in finance (BIRD), public safety (NYC Motor Vehicle Collisions), and HR analytics. Using a state-of-the-art NL2SQL agent (GPT-4o-based, instantiated within the Blue platform), the authors generated workflows for each intent and manually analyzed their outputs. Critically, only those workflows that were syntactically valid, executed successfully, but presented misaligned operationalizations were classified as failures.

Three findings structure the results:

• High frequency of operationalization failures: 153 of 236 intents (65%) produced workflows that, despite technical validity, failed to correctly instantiate the analytic concept.
• Generality across domains and benchmarks: Independent analyses on two BIRD benchmark workloads revealed similarly high operationalization failure rates (61% and 68%), demonstrating generality beyond curated workloads.
• Structurally recurring failure classes: Failures consistently clustered into a small set of operational categories, suggesting systematic architectural limitations rather than idiosyncratic model errors.

Taxonomy of Operationalization Failure Classes

The analysis identifies five principal, recurring failure classes that systematically impede semantic fidelity in workflow generation:

• Comparative Grounding: The agent operationalizes comparative concepts using arbitrary thresholds rather than contextually distributional or domain-anchored logic (e.g., "unusually large withdrawals" implemented as amount > 10000).
• Role Confusion: Analytical roles are incorrectly assigned, conflating frequency, intensity, or identification (e.g., using ORDER BY COUNT(*) as a proxy for demanding job roles).
• Process Reasoning: Temporal or process-oriented concepts are degraded into static row-level predicates, neglecting event sequence, persistence, or recurrence (e.g., "recurring overdrafts" mapped to balance < 0).
• Quantitative Reasoning: Derived quantities are constructed in a semantically invalid manner, ignoring normalization or compositional constraints (e.g., ranking intersections by total fatalities instead of fatality rate).
• Policy Grounding: Institutionally or externally defined criteria are ignored, replaced by arbitrary or locally inferred surrogates (e.g., inferring reportable incident criteria from local data statistics).

These categories are tightly coupled to the absence of accessible semantic artifacts—such as business glossaries, metric ontologies, or policy documentation—beyond the database schema and sample values.

Implications for Agentic Data Systems and Conceptual Admissibility

The study reframes evaluation for agentic data systems by emphasizing analytical admissibility: the alignment of computational operations with user-level conceptual intent. The authors demonstrate that existing agent pipeline architectures, even with advanced in-context learning and schema linking, are fundamentally incomplete if not equipped with richer semantic representations.

To address the semantic gap, two main architectural directions are discussed:

1. Direct Semantic Artifact Integration: Incorporating external knowledge sources—policies, process models, metrics definitions—could grant agents access to requisite context. However, several technical obstacles arise, including effective retrieval, disambiguation, consistency, and adaptation of heterogeneous semantic artifacts.
2. Intermediary Semantic Representations: Model architectures or orchestration layers that internalize operationalization knowledge—such as comparative baselines, process semantics, admissible operations—could guide more faithful instantiations of analytical concepts. Such intermediate representations must be explicitly incorporated and standardized for deployment in enterprise AI systems.

Theoretical and Practical Ramifications

These findings expose inherent limitations in the dominant paradigms for LLM-powered workflow generation for enterprise analytics. They highlight a class of failures that are neither addressed by improved model scale nor by better schema linkage and point to the need for future research in intermediate semantics, measurement theory-guided modeling, and explicit admissibility criteria. Practically, systems deployed for high-stakes analysis (finance, public safety) face misalignment risks that undermine the interpretability and trustworthiness of AI-automated analytics.

Conclusion

This study systematically catalogs operationalization failures encountered by agentic data systems in analytical workflow generation, establishing the existence and breadth of the semantic gap between natural language analytical intent and executable computation (2607.00828). The taxonomy of failure classes and the proposal of analytical admissibility as an evaluation axis direct future research toward richer semantic integration. Developing mechanisms for encoding and leveraging process knowledge, domain policies, and measurement logic stands as an urgent agenda for both pragmatic deployment and theoretical advancement in AI-native data systems.