Repository-Verified Agent Evaluation

Teams shipping coding agents need a commit-aware scorecard that measures what developers kept, what they threw away, and how much review friction the agent created. SWE-chat gives the clearest case for this. Across about 6,000 real sessions, agent-written code survived at 50.3% overall, with collaborative sessions dropping to 44.1%. Users pushed back in 39% of turns, and vibe-coded work carried higher token cost, slower time per committed line, and more introduced vulnerabilities than collaborative work. Those numbers support a concrete product change: add post-session attribution and commit linkage to agent logs, then rank prompts, repos, and workflows by code survival, review churn, interruption rate, and security findings per committed line. The first users are teams already paying for coding agents across shared repositories and arguing about whether the output is helping. A cheap pilot is a GitHub app or CLI wrapper that links session traces to merged diffs and shows survival and rejection rates after one week of normal use.

A short, task-specific AGENTS.md generator with an eval loop is now a practical build for teams using repository agents. The evidence is concrete enough to treat documentation as a performance input. In Augment's study, top files around 100 to 150 lines produced 10 to 15% gains, and a six-step workflow for adding a new integration cut missing wiring files from 40% to 10% while raising correctness by 25% and completeness by 20%. The same study also shows how easy it is to make things worse: architecture-heavy files pulled in about 80K irrelevant tokens and warning-only rules doubled PR time. A useful product here is not a generic doc writer. It is a repo scanner that drafts AGENTS.md around a narrow task class, inserts decision tables and small code examples from the local codebase, and runs before/after task replays on recent PRs. The first buyers are platform and developer productivity teams that already maintain internal setup docs but do not know which instructions agent harnesses actually read.

Node.js package security scanning can move one step closer to triage-ready output by generating and executing proof-of-concept exploits for suspected taint-style bugs. LLMVD.js is a clear signal that this is now buildable. The system confirms 84% of benchmark vulnerabilities, far above prior tools in the excerpt, and produced validated exploits for 36 of 260 recently released packages. Its pipeline matters as much as the headline number: it separates candidate finding, exploitability judgment, constraint inference, and execution-backed confirmation with class-specific oracles for path traversal, code injection, prototype pollution, and command injection. That supports a practical workflow change for registry operators, supply-chain security vendors, and larger application teams with many npm dependencies. Put exploit confirmation after static suspicion and before analyst review, so triage starts with packages that already have a reproducible artifact. A small validation step is to run this on a recent internal dependency set and measure how many scanner alerts can be collapsed into confirmed cases with runnable proofs.