Recoleta
GitHub
Source note

PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization

Authors
Ryan Deng, Yuanzhe Liu, Bastian Lipka, Yao Ma, Xuhao Chen, Tim Kaler, et al.
Source
arXiv
Published
Collection
Software Intelligence
Code IntelligenceAutomated Software ProductionRepository OptimizationSoftware AgentsProfiler Guided Optimization
Open arXivMarkdown source

Summary

PerfAgent improves repository-level code optimization by giving coding agents profiler summaries, iterative speedup feedback, and targeted correctness checks. On GSO and SWE-fficiency-Lite, it more than doubles the rate of patches matching expert performance compared with OpenHands using GPT-5.1.

Problem

  • General-purpose coding agents often optimize visible Python code while missing bottlenecks across abstraction layers or inside native extensions.
  • They may stop after the first passing patch and test too narrowly, producing modest speedups or correctness regressions.
  • The problem matters because repository-level optimization requires preserving behavior while approaching expert performance, not merely passing tests.

Approach

  • PerfAgent uses the low-overhead py-spy sampling profiler to identify hotspots in Python and native code, then converts raw samples into a concise summary containing locations, call context, self-time, total-time, and runtime share.
  • An objective-driven controller intercepts the agent's stop signal, rebuilds and validates each patch, re-profiles the workload, and asks the agent to continue for up to 5 iterations.
  • A best-patch selector retains the fastest correct patch rather than the last submitted patch.
  • Selective validation with pytest-testmon runs tests affected by the changes and returns build or test failures as feedback without requiring the full repository test suite.

Results

  • With GPT-5.1, the rate of expert-matching patches increased from 19.6% to 39.2% on GSO and from 26% to 74% on SWE-fficiency-Lite, relative to OpenHands.
  • On GSO, the benchmark contains 102 tasks across 10 repositories; average harmonic-mean speedup rose from 2.5x at turn 1 to 6.4x at turn 5, while validation failures fell from 53 to 5 instances.
  • PerfAgent reached low-level C, C++, Cython, or Rust code on 48 GSO instances, compared with 31 for OpenHands, including 21 cases where OpenHands changed only Python.
  • Selective testing reduced the number of tests run by 66%–99% on GSO and 47%–98% on SWE-fficiency-Lite compared with full-suite testing.
  • A traced NumPy string-replacement example reached a 5.87x speedup; the controller retained this fourth-turn patch over a slightly slower 5.85x fifth-turn patch.
  • The paper reports that PerfAgent surpassed an oracle best-of-five baseline at substantially lower cost, although the excerpt does not provide the exact cost or score values; results on SWE-fficiency-Lite later in the loop are limited by its $5 per-task budget.

Link