Intermediate Evidence Gates for Coding Agents

A coding-agent repair loop can give the model a ranked set of likely faulty lines before it asks for another edit. FLARE shows the concrete shape: a lightweight diagnostic model scores lexical units, pools them to line-level suspiciousness, prompts refinements against the top-k lines, runs tests, and keeps the candidate with the best execution outcome. With k=10, FLARE improved Pass@1 across five base models on LiveCodeBench and BigCodeBench, while its diagnostic model reached 67% Top-1 and 89% Top-10 localization accuracy on 100 LiveCodeBench tasks.

The workflow fits teams already collecting failed agent patches in CI. Store the failing program, test output, token probabilities if available, ranked suspicious lines, generated candidates, and test outcomes. Add a small behavior check for proposed edits: Neural Change Prediction reports that mutation-output pairs can train a model to predict the likely effect of a code change, with fine-tuned GPT-4.1 reaching 95% accuracy for output-change prediction on single Python mutations. A pilot can replay recent failed generations and compare execution-only repair against line-ranked candidate search on pass rate and review minutes per accepted fix.

Multi-agent coding runs should expose task dependencies before implementation starts. SPOQ gives a concrete operating pattern: split an epic into 1-4 hour tasks, build a DAG, dispatch independent tasks in parallel waves, and require planning and code validation gates before moving to the next stage. The reported gains are practical for teams with limited model concurrency: SPOQ measured a stable 1.4x speedup on a 2-slot real LLM backend and raised test pass rate from 91.25% to 99.75% with dual validation.

This is most useful for teams letting agents open pull requests from larger tickets. The adoption change is to add a human planning checkpoint before agents code, then require a scored code gate after each task. The human role is specific: approve task splits, resolve ambiguous requirements, and review risky design choices. A broader synthesis cites 456,535 agent-authored pull requests across 61,453 repositories, while also reporting lower merge rates and fewer structural changes for agent-authored pull requests. That pattern supports adding gates around agent work where mergeability and design fit matter.

Terminal-agent training data should be selected by whether the trace shows useful inspect-act-verify behavior. The terminal-agent study found that a lower-scoring teacher, DeepSeek-V3.2, trained stronger Qwen3 students than Claude Opus 4.6 because its traces exposed more environment-grounded steps. Masking observation-command links cut Targeted Observation Ratio from 13.4% to 5.3% and dropped Qwen3-32B performance from 20.6% to 13.8%.

A training team can turn this into a trace gate. Log file inspections, commands, test runs, error messages, and the path or state each command depends on. Score each successful trajectory for Targeted Observation Ratio, keep high-TOR traces at fixed data budgets, and flag traces where an action has no visible supporting observation. EvoTrainer points in the same operational direction for long-horizon software agents: its trainer reads rollouts, logs, configs, code diffs, and metrics, then updates diagnostics when current evidence cannot explain outcomes. The cheap validation is an SFT run on matched successful traces split by TOR and measured on Terminal-Bench-style tasks.