Recoleta
GitHub
Source note

Benchmarking and Evaluating VLMs for Software Architecture Diagram Understanding

Authors
Shuyin Ouyang, Jie M. Zhang, Jingzhi Gong, Gunel Jahangirova, Mohammad Reza Mousavi, Jack Johns, et al.
Source
arXiv
Published
Collection
Software Intelligence
Vision Language ModelsSoftware ArchitectureDiagram UnderstandingBenchmarkingMultimodal Reasoning
Open arXivMarkdown source

Summary

This paper introduces SADU, a benchmark for testing whether vision-language models can read and reason over software architecture diagrams. Current models are far from reliable on this task: the best reported accuracy is 70.18%, and several widely used models perform much worse.

Problem

  • Software engineering benchmarks focus mostly on code, while early design artifacts such as architecture diagrams are less studied.
  • Architecture diagrams carry system structure, behavior, and data relationships, so weak diagram understanding can break architecture-level assistance and create mismatches between design and implementation.
  • General multimodal benchmarks do not test software-specific diagram semantics in a focused way.

Approach

  • The authors build SADU, a benchmark with 154 curated software architecture diagrams: 51 behavioral, 53 structural, and 50 ER diagrams.
  • Each diagram gets manual structured annotation in JSON, including entities, relations, clusters, and optional attributes or methods; the annotation took about 160 human hours.
  • The benchmark includes 2,431 QA tasks across 24 subtypes, covering counting, retrieval, and relation reasoning over diagram elements.
  • They evaluate 11 VLMs from the Gemini, Claude, GPT, and Qwen families with temperature set to 0, using both exact-match rule-based scoring and an LLM-as-a-judge protocol.
  • They also analyze difficulty by diagram type, question subtype, complexity, and token cost.

Results

  • SADU is built from an initial pool of 1,044 diagrams, filtered down to 154 after removing 318 duplicates, 157 low-resolution images, 64 unreadable diagrams, and 311 items excluded for relevance or annotation quality.
  • Best overall accuracy is 70.18% for gemini-3-flash-preview. Next are gemini-2.5-flash: 69.68%, gemini-3.1-flash-lite-preview: 66.31%, claude-sonnet-4.5: 56.36%, gpt-5-nano: 55.45%, and gpt-4o-mini: 17.77%.
  • Performance varies by diagram type. gemini-3-flash-preview scores 63.58% on behavioral, 68.87% on structural, and 78.53% on ER diagrams. gemini-2.5-flash reaches the highest ER score reported in the excerpt at 82.54%.
  • Open-weight Qwen models trail the top closed models, with a size trend: qwen-2.5-VL-32B: 45.17%, 7B: 38.58%, 3B: 31.30% overall.
  • Diagram complexity hurts all models: accuracy drops as the number of entities and relations grows, with behavioral diagrams and grounding-heavy retrieval tasks described as the hardest cases.
  • Token cost differs sharply across models. gpt-5-nano uses 1475.19 completion tokens on average, while many lighter models stay around 13–20 tokens; gemini-3.1-flash-lite-preview is reported as a strong accuracy-cost tradeoff at 66.31% accuracy with 16.94 average completion tokens.

Link