Source note
StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows
Engineering AgentsExecutable BenchmarkArtifact TraceabilityWorkflow ValidationMultimodal Reconstruction
Summary
StructureClaw evaluates structural-engineering agents through complete, executable evidence chains rather than final answers alone. Its benchmark shows that governed skills and artifact-based workflow execution substantially improve end-to-end success, while invalid-input handling and multimodal model reconstruction remain difficult.
Problem
- Structural-engineering requests require consistent requirements, models, validation records, solver outputs, code checks, and reports; fluent text or executable-looking code can conceal missing or inconsistent artifacts.
- Existing agent evaluations usually test isolated capabilities and do not require a complete, executable, traceable workflow.
- This matters because an engineering result is only reviewable when its computational evidence and execution history support the reported conclusion.
Approach
- StructureClaw uses governed domain skills, typed tools, backend providers, and shared typed artifact state to preserve inspectable workflow outputs.
- A ReAct-style agent loop selects tools, updates artifacts, validates models, requests clarification, repairs errors, executes supported backends, or terminates safely when execution is unsupported or unsafe.
- The structural-model protocol records geometry, topology, materials, sections, supports, loads, combinations, units, and analysis metadata; OpenSeesPy provides the main open-source analysis backend.
- StructureClaw-Bench contains 150 controlled scenarios split evenly across standard workflows, interactive robustness, and multimodal structural-model reconstruction. A scenario succeeds only when every required routing, artifact, execution, interaction, and reporting assertion passes in one run.
Results
- Across 10 agent-model configurations and 50 standard cases per configuration, mean Success Rate increased from 56.8% in generic-only mode to 88.6% with the automatic workflow, a 31.8-percentage-point gain. All 10 configurations improved.
- Kimi-K2.6 achieved 100.0% on the 50 standard cases; DeepSeek-V4-Flash and GLM-5.2 each achieved 96.0%. The comparison is a system-level result because it changes skill routing, structural priors, artifact expectations, and validation guidance together.
- Automatic-mode diagnostic rates were 98.0% for structural-type matching, 100.0% for skill selection, and 90.8% for model matching.
- Interactive robustness produced a mean Success Rate of 91.0%, with configuration results ranging from 88.0% to 94.0%; pooled assertion rates were 89.6% for clarification, 90.7% for avoiding invalid models, and 91.0% for withholding analysis when appropriate.
- The reported remaining weaknesses are safe handling of invalid numerical inputs and fixture-consistent reconstruction from images or DXF files. The excerpt provides no aggregate multimodal Success Rate, so multimodal performance cannot be quantified here.