Coding agents need repository-scale tests and context defenses

DeNovoSWE treats whole-repository generation as a training problem with executable checks. Its 4,818 document-to-repository instances are built from real repositories with Docker environments, test coverage filters, sandbox cleanup, and documentation written at the capability level. Fine-tuning Qwen3-30B-A3B on the dataset raises BeyondSWE-Doc2Repo performance from 5.8% to 47.2%, a large gain on a task that requires file layout, APIs, dependencies, and cross-component behavior.

EsoLang-Bench tests a different skill: adaptation inside an unfamiliar executable interface. The strongest agents often write Python, JavaScript, or Rust generators that emit esoteric-language code, then debug those generators with local interpreters. The benchmark separates deployed agents sharply, with an 88.4 percentage-point mean-score spread across six agents. That result gives tool use and local execution a concrete role in measuring coding ability.

DeLM makes coordination the main object of evaluation. Agents claim queued subtasks, read a shared verified context, and write back compact updates about facts, failed attempts, constraints, and partial fixes. On SWE-bench Verified with Gemini 3 Flash, it reports 65.7% Avg.@1 at $0.12 per task, compared with 56.4% Avg.@1 for the strongest listed baseline and roughly double the cost for several baselines.

MASTOR applies agent coordination to REST API testing. It reads implementation source code, records endpoint constraints and response facts, then generates status, field, and cross-operation semantic oracles. Across 13 open-source RESTful API projects, it produced 10,022 oracles and reached a 75.4% average mutation score. The stronger evidence here is practical: agents are useful when their outputs are tied to source evidence and checked by a review step.

Two papers make code context itself a security target. The adversarial-context study shows that comments, documentation, variable names, and reference examples can steer generators toward vulnerable code at inference time. Across 2,800 trials, adversarial context raised mean vulnerability generation from 3.5% to 37.4%. Nearby context was especially effective: prompts placed 10 to 50 tokens before the target function reached 62.1% attack success.

The natural-backdoor paper adds a model-side risk. It studies Code Language Models (CodeLMs) across 44 scenarios and reports that normally trained models can contain trigger-like code features that bias outputs toward target labels. ScanNBT searches for diverse natural triggers by trigger inversion. The evidence is less numerically detailed than the adversarial-context study, but it broadens the concern beyond prompt manipulation to features learned during ordinary training.