Recoleta
GitHub
Source note

ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

Authors
Zhongyuan Peng, Dan Huang, Chuyu Zhang, Caijun Xu, Changyi Xiao, Shibo Hong, et al.
Source
arXiv
Published
Collection
Software Intelligence
Code IntelligenceAutomated Software ProductionCoding AgentsInteractive BenchmarkRepository Generation
Open arXivMarkdown source

Summary

ICAE-Bench evaluates whether coding agents can turn incomplete product intent into working repositories through clarification, planning, tool use, debugging, and implementation. It reports a 480-task benchmark spanning 12 languages, with experiments showing that agents often reproduce visible behavior but struggle with hidden constraints, boundary cases, and long-horizon integration.

Problem

  • Existing coding benchmarks mainly use complete specifications or existing repositories, so they do not measure whether an agent can clarify ambiguous intent and build a project from scratch.
  • This matters because interactive project building requires agents to recover requirements, preserve them during implementation, and deliver runnable software across repository-level constraints.

Approach

  • Derive each task from a verified open-source repository, create a complete GroundPRD and black-box behavioral cases, then hide selected API, edge-case, and architectural constraints to produce Fuzzy L1, L2, and L3 requirements.
  • Simulate clarification with a grounded User Agent backed by authored User Agent Data, preventing hallucinated requirements, implementation leakage, and unreproducible interaction.
  • Remove the original code and tests from a provisioned execution image, allowing agents to construct the repository from the fuzzy requirement and recovered information.
  • Score open-ended implementations with black-box Native and Enhanced tests plus diagnostics for functional correctness, semantic and API similarity, structural fidelity, design quality, and interaction quality.

Results

  • ICAE-Bench contains 480 tasks, with 40 tasks in each of 12 languages; ICAE-Bench-Lite contains 50 tasks across 10 languages.
  • The benchmark evaluates 6 coding models in 2 agent frameworks, Claude Code and OpenHands, and finds that ambiguous project generation remains challenging.
  • GroundPRD is reported as a strong upper bound; clarification recovers only part of the performance gap, and greater constraint coverage does not automatically produce a higher pass rate.
  • Agents generally reproduce visible behavior but struggle with hidden constraints, boundary cases, and long-horizon integration; the excerpt does not provide aggregate pass-rate or model-by-model metric values.
  • On ICAE-Bench-Lite, Fuzzy L1 and L3 were judged equivalent to GroundPRD for all 50 repositories, with mean semantic-similarity scores of 0.952 and 0.942, respectively, and no case judged likely to fail the same tests.

Link