Source note
Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution
Summary
This paper introduces SWE-STEPS, a dataset and evaluation framework for coding agents that work through dependent pull-request sequences instead of isolated tasks. The main claim is that standard single-PR benchmarks overstate agent ability and miss damage to long-term code quality.
Problem
- Existing coding-agent benchmarks evaluate one pull request at a time on a clean repository state, which does not match real software development where changes accumulate.
- That setup misses spillover effects from earlier agent-written code, including regressions, growing test obligations, technical debt, and higher code complexity.
- This matters because an agent that passes isolated tasks can still fail across a multi-step development sequence and leave the repository harder to maintain.
Approach
- The authors build an automated framework that mines git history to extract chains of related PRs, their metadata, changed symbols, and associated tests.
- They create SWE-STEPS, a dataset with 168 tasks and 963 PRs across 6 Python repositories, with task chains of 3 to 11 PRs.
- Each task includes an initial repo state, an ordered sequence of PR requests, and verification suites split into FAIL_TO_PASS tests for new functionality and PASS_TO_PASS tests for regression checks.
- They evaluate agents in three settings: Individual PR (isolated, SWE-bench-style reset), Global Memory / conversational coding (state persists across PRs), and PRD-based coding (all requirements given up front, final accumulated test suite checked at the end).
- Evaluation covers both functional success and repository health, using static analysis metrics such as cognitive complexity and technical debt compared with the human-written ground truth.
Results
- Dataset scale: 168 tasks, 963 PRs, 6 repositories; chains have 3 to 11 PRs.
- Task complexity: average issue text length is 3,656 words versus 195.1 for SWE-Bench and 239.8 for SWE-Gym; average edited files are 17.1 versus 1.7 and 2.5.
- Performance inflation: isolated PR evaluation overstates success by up to 20 percentage points. Example from the Mini split: Claude Sonnet 4.5 drops from 66.25% in the Individual setting to 43.75% in a continuous setting.
- On the Lite split, Gemini 3 Flash drops from 56.52% in the Individual setting to 36.59% in the Global setting.
- Across tested LLMs, introducing stateful multi-step evaluation reduces performance by about 15% to 25% relative to isolated evaluation.
- The paper also claims agents worsen repository health versus human developers by producing code with higher cognitive complexity and more technical debt, measured with SonarQube, but the excerpt does not provide exact numeric deltas for those metrics.