Recoleta
GitHub
Source note

Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Dataset

Authors
Mahmoud Abujadallah, Ali Arabat, Mohammed Sayagh
Source
arXiv
Published
Collection
Software Intelligence
AI AgentsPull RequestsBug FixingCode ReviewSoftware Engineering
Open arXivMarkdown source

Summary

This paper studies why AI-agent pull requests that try to fix bugs get rejected, using 306 rejected fixes from the AIDev dataset. It matters because 46.41% of fixes from Copilot, Devin, Cursor, and Claude were rejected, which wastes review time, tests, tokens, and compute.

Problem

  • AI coding agents often open fix PRs that developers later close without merging.
  • The paper asks why these bug-fix PRs fail and how much review effort they waste.
  • The goal is to make agent-generated fixes easier to trust and cheaper to review.

Approach

  • The authors use the AIDev dataset, filter to bug-fix PRs created or co-authored by Copilot, Devin, Cursor, and Claude, and sample 306 rejected PRs from 1,497 rejected fixes.
  • Two authors manually label each PR by reading discussions, CI results, and related artifacts, then reconcile disagreements with a third author.
  • They build a rejection taxonomy with 14 reasons grouped into four high-level themes.
  • They measure wasted effort with code churn and comment counts for each rejection category.
  • They report inter-rater agreement with Cohen’s κ = 0.605.

Results

  • The dataset contains 3,225 fix PRs from the four agents, and 1,497 were rejected, a rejection rate of 46.41%.
  • The manual study finds 14 rejection reasons in four themes: relevance, implementation issues, provider failure, and technical issues.
  • Relevance problems are the most common visible reason, led by inactivity at 53 cases (17.3%) and superseded PRs at 18 cases (5.9%); low priority appears in 3 cases (1.0%).
  • Implementation issues include incorrect fixes (17 cases, 5.6%) and wrong approaches (8 cases, 2.6%); CI failures account for 21 cases (6.9%).
  • Rejected PRs have median code churn between 81 and 293 lines, with technical issues showing the highest churn and implementation issues around a median of 103 lines.
  • Rejected PRs receive a median of 1 to 4.5 comments, similar to merged fixes, so a large share of review discussion goes into changes that are later discarded.

Link