Recoleta
GitHub
Source note

How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions

Authors
Ningzhi Tang, Chaoran Chen, Gelei Xu, Yiyu Shi, Yu Huang, Collin McMillan, et al.
Source
arXiv
Published
Collection
Software Intelligence
Coding AgentsDeveloper Agent MisalignmentCode IntelligenceHuman AI InteractionSoftware Engineering
Open arXivMarkdown source

Summary

This paper finds that coding-agent failures in real developer sessions are mostly alignment failures that force developers to correct, verify, or regain control. It analyzes 20,574 IDE and CLI sessions and reports 16,118 evidence-grounded misalignment episodes.

Problem

  • Coding agents now edit files, run commands, and manage multi-turn tasks, so failure is not limited to wrong code output.
  • Existing benchmark trace analyses miss how developers detect and correct agent behavior during real work.
  • The problem matters because most visible failures create developer effort and trust costs, even when they do not permanently damage the project.

Approach

  • The authors define misalignment as a breakdown visible through developer correction or pushback in chat logs.
  • They combine two datasets: 14,789 SpecStory sessions and 5,785 SWE-chat sessions, totaling 20,574 sessions across 1,639 repositories.
  • They use GPT-5.4 to extract candidate misalignment episodes from whole sessions, then run a second evidence-checking pass to remove unsupported claims.
  • They retain 16,118 validated episodes from 29,896 extracted candidates, with estimated precision of 0.93 and mean recall rating of 1.77 out of 2.00.
  • Each episode is labeled on symptom, cause, outcome, and resolution; human expert agreement is 0.83 and LLM judge accuracy is 0.81.

Results

  • The largest symptom category is developer constraint violation at 38.33% of episodes, followed by misread developer intent at 26.95%, inaccurate self-reporting at 22.58%, and faulty implementation at 17.82%.
  • The largest cause category is instruction-following failure at 36.49%; 73.68% of developer constraint violations are attributed to this cause.
  • 90.50% of episodes impose effort or trust cost only, while 8.44% cause easily reversed system damage and 0.07% cause hard-to-reverse system damage.
  • Among system-damage cases (n=1,372), 75.80% affect code or task state, 18.51% affect project state, 2.11% affect environment or configuration, and 3.57% affect external state.
  • Only 9.33% of episodes show visible resolution in the logs; among resolved cases (n=1,504), 91.49% require explicit developer pushback, 2.99% are agent self-corrections, and 5.52% end with developer takeover.
  • CLI sessions show more constraint violations than IDE sessions, 49.49% versus 32.26%; IDE sessions show more faulty implementation, 22.89% versus 8.49%.

Link