Recoleta
GitHub
Source note

Understanding Agents: Code Coverage for Coding Agents

Authors
matt_d
Source
Hacker News
Published
Collection
Software Intelligence
Coding AgentsCode CoverageSecurity AuditingAgent ObservabilityDeveloper Tools
Open originalMarkdown source

Summary

This post introduces an open-source tool that reconstructs and visualizes which code a coding agent read during an audit run. It helps humans compare agent behavior across prompts, models, and reasoning levels, and inspect what the agent looked at and missed.

Problem

  • Coding agents can report bugs after long audit runs, but current tools give weak visibility into which files and lines the agent actually inspected.
  • That gap makes it hard to judge whether the agent searched the relevant attack surface, compare runs fairly, or guide follow-up audits.
  • For security auditing, this matters because an agent cannot find bugs in code it never read, and humans need a way to spot blind spots.

Approach

  • The tool parses local .jsonl session logs from Claude Code and codex-cli, which contain prompts, reasoning summaries, and executed tool commands.
  • It converts the agent's file-reading actions into covered line ranges and links those ranges to the sub-task or intent associated with the read.
  • It shows the result in a web UI with highlighted covered lines, a project treemap, per-line coverage counts, and the sub-tasks that touched each region.
  • The tool is meant for post-hoc inspection: humans can compare prompts, models, reasoning effort, and repeated runs, then direct later audits toward uncovered areas.

Results

  • On an OpenSSH pre-auth RCE audit task, GPT-5.4 runs stayed focused on a narrower part of the obvious pre-auth surface across five runs per setting.
  • Median uniquely covered lines for GPT-5.4 increased with reasoning budget: about 8.3k for gpt5.4-medium, 13.5k for gpt5.4-high, and 17.7k for gpt5.4-xhigh.
  • Files touched per GPT-5.4 run also increased with reasoning effort: roughly 24, 35, and 40 files for medium, high, and xhigh.
  • Opus 4.6 covered much more code: median uniquely covered lines were about 31.8k for opus4.6-high and 30.3k for opus4.6-medium, with one medium run reaching roughly 50k lines.
  • Across five runs, Opus configurations touched 298 distinct files for high and 395 for medium, while GPT-5.4 settings stayed in the 40 to 62 range.
  • The authors state that coverage is not a direct bug-finding metric, but repeated runs still matter because agent behavior is probabilistic and can surface different bugs over time.

Link