Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows

Source

arXiv

arxiv.org/abs/2607.14896v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Sizhong Qin, Yi Gu, Yao Jiang, Ao Cai, Changjian Zhou, Shaoxuan Shuai, Jiachang Wang, Tianhao Shen, Yueqiang Li, Xinhao Li, Li Zeng, Yueshi Chen, Dachen Gao, Genrong Xu, Wenjie Liao, Xinzheng Lu
  • Venue: arXiv: cs.SE
  • Published: 2026-07-16
  • Collected: 2026-07-17
  • URL: http://arxiv.org/abs/2607.14896v1
  • PDF: https://arxiv.org/pdf/2607.14896v1
  • DOI: UNKNOWN

Plain-Language Brief

Researchers introduce StructureClaw, a workbench designed to evaluate large language model (LLM) agents in structural engineering tasks. This workbench ensures the LLMs follow structured workflows and produces complete evidence chains for these tasks. The researchers also present StructureClaw-Bench, an executable benchmark of 150 scenarios used to test the LLMs' ability to execute standard workflow cases, handle invalid numerical inputs, and reconstruct multimodal structural models. The study finds that artifact-centered evaluation can help identify failures in workflows that may not be apparent from final responses alone.

What the Document Says

  • Researchers present StructureClaw, a workbench for evaluating LLM agents in structural engineering tasks.
  • The workbench ensures the LLMs follow governed engineering skills and produce complete evidence chains.
  • StructureClaw-Bench is an executable benchmark of 150 scenarios used to test LLMs' standard workflow execution, interactive robustness, and multimodal structural model reconstruction.
  • The study compares ten agent-model configurations on the same 50 standard cases, with the average Success Rate rising from 56.8% with the generic-skill baseline to 88.6% with the full automatic workflow.
  • The study identifies two main challenges for LLMs: safe handling of invalid numerical inputs and fixture-consistent reconstruction of structural models.
  • Artifact-centered evaluation can help identify failures in workflows that may not be apparent from final responses alone.

Key Concepts Mentioned

  • StructureClaw
  • LLM agents
  • Structural Engineering Workflows
  • Scenario
  • Success Rate
  • artifact-centered evaluation
  • invalid numerical inputs
  • fixture-consistent reconstruction

Claims Made by the Document

  • StructureClaw improves the Success Rate of LLM agents in structural engineering tasks.
  • Artifact-centered evaluation helps identify failures in workflows that may not be apparent from final responses alone.

Evidence Used

  • Average Success Rate rising from 56.8% with the generic-skill baseline to 88.6% with the full automatic workflow

Methods / Approach

  • Evaluation of LLM agents on structured engineering tasks using StructureClaw and StructureClaw-Bench

Limitations Stated by the Authors

  • Unknown, as the text does not mention any limitations explicitly stated by the authors.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The study only evaluates ten agent-model configurations on the same 50 standard cases. It may not represent the performance of other LLM agents or scenarios beyond the ones tested.
  • The results are based on an executable benchmark, which may not fully capture real-world structural engineering problems.

Open Questions

  • What is the performance of StructureClaw and the evaluated LLM agents in real-world structural engineering scenarios?
  • How can the identified challenges with safe handling of invalid numerical inputs and fixture-consistent reconstruction be addressed?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The text does not provide specific details on the nature of these tasks or the types of LLM agents being evaluated.

Raw Abstract / Excerpt

Addressing a structural-engineering request requires more than a single answer; it requires a chain of interdependent artifacts: interpreted requirements, a computable model, validation records, solver outputs, code-check records, and a final report. Evaluations centered on question answering or script generation rarely verify this complete evidence chain and may therefore reward fluent outputs even when the underlying engineering workflow is incomplete, internally inconsistent, or non-executable. To address this limitation, we present StructureClaw, an artifact-centered workbench in which LLM agents operate through governed engineering skills, typed tools, shared artifact state, and local analysis backends. We also introduce StructureClaw-Bench, an executable benchmark of 150 controlled scenarios spanning standard workflow execution, interactive robustness, and multimodal structural-model reconstruction. A scenario succeeds only when all required artifact- and execution-level assertions pass in a single run. Across ten agent-model configurations, each evaluated on the same 50 standard cases, the average Success Rate rises from 56.8% with the generic-skill baseline to 88.6% with the full automatic workflow. The interactive and multimodal evaluations identify two prominent remaining challenges: safe handling of invalid numerical inputs and fixture-consistent reconstruction of structural models. These findings show that artifact-centered evaluation can expose workflow-level failures that are difficult to identify from final responses alone, providing a more rigorous basis for evaluating and improving structural-engineering agents. The code and benchmark are available at https://github.com/structureclaw/structureclaw.