Source note2 min read
Exploring the Semantic Gap in Agentic Data Systems: A Formative Study of Operationalization Failures in Analytical Workflows
Source
arXiv
arxiv.org/abs/2607.00828v1
Source
- Source name: arXiv
- Source kind: paper
- Authors / organization: Jalal Mahmud, Eser Kandogan
- Venue: arXiv: cs.DB
- Published: 2026-07-01
- Collected: 2026-07-02
- URL: http://arxiv.org/abs/2607.00828v1
- PDF: https://arxiv.org/pdf/2607.00828v1
- DOI: UNKNOWN
Plain-Language Brief
A study found that large language models (LLMs) often struggle to operationalize analytical concepts due to a semantic gap between user-level analytical concepts and the information available to workflow-generation systems, leading to frequent failures in generating accurate queries and workflows across various domains.
What the Document Says
- Large language models are used for query generation, tool invocation, and analytical workflow creation.
- Despite advancements in workflow generation and execution, operationalizing analytical concepts can be challenging due to a lack of semantic information.
- The study identified 153 recurring failures across finance, human resources, and public safety domains despite successful workflow generation and execution.
- Five classes of failures were identified: comparative grounding, process reasoning, quantitative reasoning, role confusion, and policy grounding.
- These findings indicate a semantic gap between user-level analytical concepts and the information available to workflow-generation systems.
- The study raises questions about the admissibility of analytical operations and suggests that future agentic data systems may require richer semantic representations.
Key Concepts Mentioned
- large language models
- analytical workflows
- semantic gap
- operationalization failures
- comparative grounding
- process reasoning
- quantitative reasoning
- role confusion
- policy grounding
Claims Made by the Document
- Large language models struggle to operationalize analytical concepts due to a semantic gap.
- This struggle results in frequent failures across various domains.
- Future agentic data systems may require richer semantic representations.
Evidence Used
- The study identified 153 recurring failures across three domains.
Methods / Approach
UNKNOWN
Limitations Stated by the Authors
UNKNOWN
Limitations Noted by the Agent
Agent interpretation — not from the source.
UNKNOWN
Open Questions
UNKNOWN
Notes for Future Q&A
May include interpretation. Labeled where it does.
UNKNOWN
Raw Abstract / Excerpt
Large language models (LLMs) are increasingly used to generate queries, invoke tools, and construct analytical workflows. Although recent advances have substantially improved workflow generation and execution, the semantic information required to operationalize analytical concepts often lies beyond what is explicitly represented in database schemas and data values. We present a cross-domain formative study of operationalization failures in agent-generated analytical workflows. Across 236 analytical intents spanning finance, human resources, and public safety domains, we identify 153 recurring failures despite successful workflow generation and execution. Our analysis reveals five recurring classes of failures: comparative grounding, process reasoning, quantitative reasoning, role confusion, and policy grounding. These findings suggest a semantic gap between user-level analytical concepts and the information available to workflow-generation systems. More broadly, they raise questions about the admissibility of analytical operations and suggest that future agentic data systems may require richer semantic representations to bridge the gap between analytical intent and executable computation.
