Source note
SWE-chat: Coding Agent Interactions From Real Users in the Wild
Summary
SWE-chat is a large dataset of real coding-agent sessions from open-source developers using public GitHub repositories. The paper uses it to measure how people actually work with coding agents, how much agent-written code survives, and where these systems fail in normal development workflows.
Problem
- Research on coding agents relies heavily on curated benchmarks, but those benchmarks miss real user prompts, iterative back-and-forth, tool use, and what code developers actually keep.
- Without in-the-wild data, it is hard to measure practical usefulness, failure modes, human oversight, cost, or security impact.
- This matters because coding agents are already used at scale, and product or model decisions based only on benchmark results can misread real workflow performance.
Approach
- The authors build SWE-chat, a living dataset collected from developers who opt into logging via the Entire.io CLI on public GitHub repositories.
- The dataset links full session transcripts to commits, including user prompts, agent responses, tool calls, token usage, code diffs, and line-level human-versus-agent authorship.
- At the time of writing, SWE-chat contains about 6,000 sessions, 63,000 user prompts, 355,000 agent tool calls, 13,000 checkpoints, and 2.7 million logged events across 200+ repositories.
- The paper adds annotations for session success, user persona, prompt intent, and user pushback, using validated LLM judges plus raw logs and code-attribution metrics such as code survival, cost per committed line, time per committed line, and Semgrep-based security findings.
- The core idea is simple: record real human-agent coding sessions end to end, then compare what the agent produced with what the developer actually committed and how the developer reacted during the session.
Results
- SWE-chat is the first dataset in the comparison table that combines real human prompts, agent tool trajectories, code diffs, and code authorship attribution.
- Coding behavior is strongly split: 22.7% of sessions are human-only, 36.5% collaborative, and 40.8% are vibe coding where the agent writes at least 99% of committed code. Over a three-month window, vibe coding grew from 20% to over 40% of sessions.
- Usage goes beyond patch writing: the most common specific prompt intent is understanding code (19.0%); creating new code and git operations are each 13.4%, and debugging is 13.0%. About one third of agent tool calls are bash commands.
- Agent-written code is often discarded: overall 44.3% coding efficiency and 50.3% code survival rate; in collaborative sessions these drop to 38.2% and 44.1%, while vibe coding reaches 59.0% and 64.6%.
- Vibe coding is more expensive and slower per committed output: median 204K tokens per 100 committed lines, about 3x collaborative mode; median dollar cost is $0.13 per 100 lines versus $0.05 collaborative and $0.07 human-only; median time is 12.6 minutes per 100 lines versus 4.8 collaborative and 8.6 human-only.
- Security outcomes are worse for vibe coding: introduced vulnerabilities are 0.76 per 1,000 committed lines, versus 0.14 for collaborative and 0.08 for human-only, measured with Semgrep. The paper also reports that users push back often: 39% of turns overall, with user interruptions in roughly 3.3% to 6.0% of turns, while agents ask for clarification in only 1.1% to 2.6% of turns.