Executable feedback is outperforming prompt-only coding workflows

Repository-level optimization improves when agents receive measured runtime evidence rather than broad instructions. PerfAgent identifies hotspots with a sampling profiler, validates each patch, and feeds speedup results back for up to five rounds. With GPT-5.1, expert-matching patches rose from 19.6% to 39.2% on GSO and from 26% to 74% on SWE-efficiency-Lite.

MoST supplies a complementary form of executable guidance. It converts optimization knowledge from commits and technical documents into validated Semgrep rules, including strategies transferred across languages and architectures. On 351 historical tasks, it produced 24.44%–180.00% more exact developer-matching patches than SemOpt. Together, the studies support task-specific measurement and validated rules as stronger optimization signals than unguided generation.

Test generation is being judged by whether it separates correct patches from plausible failures, not merely whether a test runs. CoHarden challenges generated bug-reproduction tests with semantic mutations. Rigorous tests raised repair resolution by 8.5 points, while lax tests produced no gain and misaligned tests reduced resolution by 3.6 points.

CATGen addresses an earlier reliability boundary: generated unit tests must compile in dependency-heavy projects before coverage matters. It retrieves project and framework context, creates scaffolding deterministically, and applies static-analysis repairs. Across eight industrial projects, compilation success improved by 24.72%–38.05%, while token use fell by 66.83%–83.86%. The shared result is that test artifacts become more useful when deterministic checks surround model generation.

Two systems narrow generation by assembling the surrounding software context first. AutoGlue interprets each behavior-driven development step within its scenario, then retrieves related specifications and project APIs. It improved API F1 by 58.7% over few-shot prompting, although only 46.1% of outputs were directly usable.

CATGen similarly retrieves class structure, external calls, and testing frameworks before asking the model to complete a fixed test skeleton. Its industrial results show that this division of labor can improve compilation while reducing repair loops. The evidence is limited to Java-oriented testing and glue-code tasks, so broader repository editing remains unproven.