Source note
Scaling Test-Time Compute for Agentic Coding
Summary
This paper improves agentic coding by spending test-time compute on multiple full agent rollouts, then selecting and reusing the useful parts of those attempts. The main idea is to replace long noisy trajectories with short structured summaries that support both better selection and better refinement.
Problem
- Agentic coding tasks produce long action-observation traces across many terminal steps, so standard test-time scaling methods from math or single-shot code generation do not transfer well.
- Raw trajectories are hard to compare and hard to reuse because they mix useful clues with repeated logs, errors, and dead ends.
- This matters because benchmarks such as SWE-Bench Verified and Terminal-Bench v2.0 depend on long-horizon debugging and editing, where a better way to spend inference compute can raise pass rates without changing the base model.
Approach
- The method converts each rollout into a compact structured summary that records the key diagnosis, attempted fixes, progress, and failure modes.
- For parallel scaling, it uses Recursive Tournament Voting (RTV): run N=16 rollouts, summarize them, compare summaries in small groups, and repeat until one rollout remains. The default setup uses pairwise groups (G=2) and V=8 comparison votes.
- For sequential scaling, it adapts Parallel-Distill-Refine (PDR) to agentic coding: build a refinement context from K=4 summaries from the previous iteration, then launch a fresh new rollout in a reset environment conditioned on those summaries.
- The full pipeline is: iteration-0 rollouts, RTV select-top-K, iteration-1 refined rollouts, then final RTV selection.
- Ablations show that structured summaries work better than full traces for selection, pairwise recursive comparisons work better than flat many-way comparisons, and refining from multiple prior summaries works better than refining from one.
Results
- Main full-pipeline gains on SWE-Bench Verified: Claude-4.5-Opus 70.94% → 77.60% (+6.66), Gemini-3.1-Pro 72.25% → 76.60% (+4.35), Claude-4.5-Sonnet 67.41% → 75.60% (+8.19), Gemini-3-Flash 70.79% → 76.00% (+5.21), GPT-5-0825 61.41% → 69.80% (+8.39).
- Main full-pipeline gains on Terminal-Bench v2.0: Claude-4.5-Opus 46.95% → 59.09% (+12.14), Gemini-3.1-Pro 52.49% → 64.77% (+12.28), Claude-4.5-Sonnet 40.62% → 56.82% (+16.20), Gemini-3-Flash 37.93% → 48.86% (+10.93), GPT-5-0825 31.32% → 38.64% (+7.32).
- RTV alone already helps. Example: Claude-4.5-Sonnet improves on SWE-Bench Verified 67.4% → 73.6% and on Terminal-Bench v2.0 40.6% → 54.6% with N=16, G=2, V=8.
- In sequential refinement on 100 sampled SWE-Bench Verified tasks, Gemini-3.1-Pro improves from 72.69% → 73.75% with single-rollout refinement, 72.69% → 76.94% with random-K refinement, and 72.69% → 79.25% with RTV-selected K=4 summaries. Claude-4.5-Sonnet reaches 78.06% with select-K refinement.
- Context quality predicts next-iteration success. With K=4 selected prior rollouts, iteration-1 pass rates rise to about 97.3% for Claude-4.5-Sonnet and 99.7% for Gemini-3.1-Pro when the refinement context contains 4/4 passing rollouts.
- Sequential refinement also reduces average agent steps by about half in Table 4 while improving pass@1, and the paper states the method solves some tasks that none of the initial 16 rollouts solved.