Agent workflow observability

Teams shipping coding agents need a post-merge review lane that measures what survives after merge, not just whether the agent opened a successful pull request. A practical build is a repository report that tags agent-authored PRs, tracks later edits on the touched lines and files for 7, 30, and 90 days, and feeds the results back into review policy. Start with repositories that already label Codex, Claude Code, Copilot, Jules, or Devin activity. Flag patterns such as repeated follow-up fixes, high rewrite rates in generated tests, or large refactors that collapse soon after merge. The point is operational: maintainers need a way to see which agent workflows create cleanup work for humans.

The evidence now supports this as a normal engineering metric. A GitHub-scale study built a dataset of 111,969 PRs across five coding agents and found that agent-authored code sees more later churn than human-authored code. A separate evaluation paper argues that software-agent results are hard to compare when work stops at final task scores and hidden runs. Joining those two lines of work yields a concrete workflow: store the run metadata for each agent PR, then compare survival and churn by model, scaffold, repository type, and review path. A cheap first check is one quarter of data from a single org with bot-authored PR detection and a simple line-survival calculation.

Agent teams should add a mandatory trace package to every benchmark or internal bake-off: exact model version, prompts, temperature, tool settings, and a summarized Thought-Action-Result log for each run. This is a buildable support layer for evaluation, not a research nicety. Without it, failures cannot be inspected, baseline claims are hard to verify, and teams end up rerunning expensive experiments to answer basic questions about why one scaffold beat another.

The current evidence points to a narrow, useful format. One review of 18 software-engineering agent papers found that only 1 compared against a relevant agentic baseline. The same paper recommends publishing TAR trajectories or summaries so later analysis can recover concrete failure modes and verification behavior. STITCH adds a second reason to keep richer traces: its gains come from filtering long runs down to decision-critical segments, with up to 63.16% relative improvement on SWE-bench Verified and 61.31% compilation pass rate on HarmonyOS ArkTS using less than 1K training trajectories. That combination makes a small internal tool credible: collect runs once, compress them into searchable decision segments, and use the same store for both evaluation review and later fine-tuning.

Personal agents with file, email, browser, and shell access need an attachment and workspace quarantine layer before prompt-injection defenses inside the model. The concrete build is a pre-execution gate that classifies incoming email, downloaded files, web content, and local skill files by trust level, strips instruction-like content from low-trust sources, and requires explicit approval before those sources can influence tool calls or credential access. Teams can test it in sandboxes with seeded malicious documents and long-context sessions.

This is now an operational requirement for high-privilege agents. ClawSafety runs 2,520 trials across 120 attack scenarios and reports attack success rates from 40.0% to 75.0% depending on model, with skill-file injection averaging 69.4% and scaffold choice changing outcomes for the same model. The supporting case study on Claude Code adds a related exposure path: minified JavaScript bundles can leak prompts, endpoints, telemetry names, and environment variables in plain text, with 147,992 strings extracted from a 13MB bundle in 1.47 seconds. If agent builders assume hostile instructions and internal logic are both easy to recover, the safer default is to isolate untrusted content before the agent reads it and to move sensitive logic out of client-visible code.