Coding-agent research is getting judged by kept code, control surfaces, and executable proof

Real usage evidence is getting harder to ignore. SWE-chat logs about 6,000 sessions across 200+ repositories and shows that coding agents are used for code understanding, git operations, debugging, and full code writing, not just benchmark-style patching. The useful correction is that output volume is a weak proxy for value. Agent-written code survives at 50.3% overall, and collaborative sessions fall to 44.1%. The same paper also shows clear cost and risk differences: vibe-coded work uses 204K tokens per 100 committed lines and introduces 0.76 vulnerabilities per 1,000 committed lines, versus 0.14 in collaborative sessions. That fits the recent run of papers that add execution and behavioral checks, but this one adds the missing in-the-wild layer: what developers kept, what they rejected, and how often they pushed back.

The control surface around coding agents is getting more explicit. One line of work treats project documents as direct performance inputs. The AGENTS.md study reports 10 to 15% gains from short, task-specific files, a drop in missing wiring files from 40% to 10% with a six-step workflow, and large failures when docs are bloated or warning-heavy. Another line treats the harness itself as the optimization target. HARBOR shows that manual flag stacking can backfire: a baseline at 15/89 tasks improved to 17/89 with one bundle of native flags, then fell to 13/89 after adding self-evaluation and 12/89 after adding more published techniques. Shift-Up sits on the process side of the same theme. It uses requirements, architecture records, and executable tests to constrain agent work, but its evidence is still qualitative. The common emphasis is clear: teams are tuning the wrapper around the model as much as the model output.

Execution-backed evaluation keeps spreading into adjacent coding tasks. WebGen-R1 uses reinforcement learning for multi-page website generation, but the key detail is its reward design: a site must pass structure checks, build, serve, and render before visual scoring matters. On its reported benchmark, valid render ratio rises from 30.56% to 95.89%, while functional quality moves from 1.59% to 29.21%. Security work is using the same pattern of concrete confirmation. LLMVD.js does not stop at suspicious flows in Node.js packages; it generates and runs proof-of-concept exploits, confirming 84% of benchmark vulnerabilities and producing validated exploits for 36 of 260 recent packages. A separate OSV-based evaluation paper makes a narrower point: even scanner comparisons need version-level ground truth before numbers are trustworthy. The throughline is practical verification. Systems are being judged by what runs, renders, or exploits under direct checks.