Source note
Does Pass Rate Tell the Whole Story? Evaluating Design Constraint Compliance in LLM-based Issue Resolution
Summary
This paper argues that test pass rate misses a large part of patch quality in LLM-based issue resolution. It introduces SWE-Shield, a benchmark that measures whether generated patches follow project-specific design constraints mined from real pull requests.
Problem
- Existing issue-resolution benchmarks mainly score patches by test pass rate, but accepted patches also need to follow project design constraints such as architecture choices, error handling, API consistency, and maintainability rules.
- Many of these constraints are not encoded in tests and are only implied in pull request review discussions, so a test-passing patch can still be unacceptable in real development.
- Without a way to extract, link, and verify these constraints, current benchmark scores overstate how useful LLM agents are for real repository work.
Approach
- The paper builds SWE-Shield, a design-aware benchmark aligned with SWE-bench-Verified and SWE-bench-Pro, covering 495 issues, 1,787 validated constraints, 6 repositories, and 10,885 extracted constraints before validation.
- It introduces DesignHunter, a two-stage LLM pipeline: first it extracts atomic design suggestions from code review threads, then it groups and synthesizes them into structured design constraints with problem, options, conditions, and reference code.
- To connect constraints to issues, the pipeline uses explicit traceability and semantic matching, followed by manual validation.
- For evaluation, it adds an LLM-based verifier that checks whether a generated patch satisfies the linked design constraints, since these constraints are non-executable and often context-dependent.
Results
- On SWE-Shield-verified, agents reach 70.25%–75.95% Pass Rate, but design satisfaction rate (DSR) is only 32.64%–50.20%.
- On SWE-Shield-pro, Pass Rate reaches 42.69%, while design violations remain high, with design violation rate (DVR) up to 45.85%.
- Functional correctness has little statistical relation to design compliance in most settings: Cramér’s V ≤ 0.11, and the paper reports no significant association in most chi-square tests.
- Across foundation models under the same swe-agent setup on SWE-Shield-pro, DSR changes by only 12 percentage points, even when Pass Rate differs much more, which suggests many design failures are shared across models.
- Adding issue-specific design guidance lowers violations, with DVR decreasing by up to 6.35 percentage points, but remaining violation rates still stay above 30%.
- The main claim is that test-based evaluation materially overestimates patch quality, because fewer than half of resolved issues are fully design-satisfying in many settings.