StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Sizhong Qin, Yi Gu, Yao Jiang, Ao Cai, Changjian Zhou, Shaoxuan Shuai, Jiachang Wang, Tianhao Shen, Yueqiang Li, Xinhao Li, Li Zeng, Yueshi Chen, Dachen Gao, Genrong Xu, Wenjie Liao, Xinzheng Lu
• Venue: arXiv: cs.SE
• Published: 2026-07-16
• Collected: 2026-07-17
• URL: http://arxiv.org/abs/2607.14896v1
• PDF: https://arxiv.org/pdf/2607.14896v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers have developed StructureClaw, a workbench that allows large language model (LLM) agents to operate within a structured engineering environment through governed skills, typed tools, shared artifact state, and local analysis backends. The study also presents StructureClaw-Bench, an executable benchmark of 150 controlled scenarios for evaluating the success rate of these LLM agents in completing standard workflow tasks, interactive robustness tests, and multimodal structural model reconstructions. The results show a significant increase in the success rate when using StructureClaw compared to a generic-skill baseline.

What the Document Says

• Researchers developed StructureClaw, an artifact-centered workbench that allows LLM agents to operate within governed engineering skills, typed tools, shared artifact state, and local analysis backends.
• StructureClaw-Bench is an executable benchmark of 150 controlled scenarios for evaluating the success rate of these LLM agents in completing standard workflow tasks, interactive robustness tests, and multimodal structural model reconstructions.
• The study includes ten agent-model configurations, each evaluated on the same 50 standard cases.
• The average Success Rate rises from 56.8% with the generic-skill baseline to 88.6% with the full automatic workflow in StructureClaw.
• Interactive and multimodal evaluations identify two prominent remaining challenges: safe handling of invalid numerical inputs and fixture-consistent reconstruction of structural models.
• The findings suggest that artifact-centered evaluation can expose workflow-level failures that are difficult to identify from final responses alone.

Key Concepts Mentioned

• Large Language Model (LLM) agents
• StructureClaw
• StructureClaw-Bench
• Structural Engineering Workflows
• Artifact-centered evaluation
• Success Rate
• Standard Workflow Tasks
• Interactive Robustness Tests
• Multimodal Structural Model Reconstructions

Claims Made by the Document

• StructureClaw improves the success rate of LLM agents in completing standard workflow tasks compared to a generic-skill baseline.
• Artifact-centered evaluation can expose workflow-level failures that are difficult to identify from final responses alone.

Evidence Used

• The average Success Rate rises from 56.8% with the generic-skill baseline to 88.6% with the full automatic workflow in StructureClaw.
• Interactive and multimodal evaluations identify two prominent remaining challenges.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Addressing a structural-engineering request requires more than a single answer; it requires a chain of interdependent artifacts: interpreted requirements, a computable model, validation records, solver outputs, code-check records, and a final report. Evaluations centered on question answering or script generation rarely verify this complete evidence chain and may therefore reward fluent outputs even when the underlying engineering workflow is incomplete, internally inconsistent, or non-executable. To address this limitation, we present StructureClaw, an artifact-centered workbench in which LLM agents operate through governed engineering skills, typed tools, shared artifact state, and local analysis backends. We also introduce StructureClaw-Bench, an executable benchmark of 150 controlled scenarios spanning standard workflow execution, interactive robustness, and multimodal structural-model reconstruction. A scenario succeeds only when all required artifact- and execution-level assertions pass in a single run. Across ten agent-model configurations, each evaluated on the same 50 standard cases, the average Success Rate rises from 56.8% with the generic-skill baseline to 88.6% with the full automatic workflow. The interactive and multimodal evaluations identify two prominent remaining challenges: safe handling of invalid numerical inputs and fixture-consistent reconstruction of structural models. These findings show that artifact-centered evaluation can expose workflow-level failures that are difficult to identify from final responses alone, providing a more rigorous basis for evaluating and improving structural-engineering agents. The code and benchmark are available at https://github.com/structureclaw/structureclaw.