Executable evidence is setting the bar for coding and agent research

Evaluation work is getting closer to full agent behavior. The strongest example is a sim/eval stack for agentic systems that tests multi-step actions, tool use, backend state, and user-visible outcomes in one run. It separates scenario design, simulation, and grading, then logs world state, tool traces, transcripts, and surface artifacts. CUJBench applies the same demand for grounded evidence to failure diagnosis. It freezes browser and backend evidence into reproducible incident snapshots, and current agents still reach only 19.7% accuracy. The shared lesson is simple: transcript scoring is too weak for systems that act through tools and changing state.

Repository-level coding still fails on basic software gates. RAT targets environment setup, a common reason code agents cannot even start, and reports much higher environment setup success on its 2,000-plus repository benchmark, including 63.2% on Python and 98.7% on Rust. A separate study of 212 real commits in eight C projects shows why this matters after setup: generated patches still fail to compile, trip static analysis, and miss tests, even when the original commits are small. This keeps execution, buildability, and repository context at the center of coding-agent research.

Two papers push code assistance into narrower, more concrete settings. MOSAIC tackles scientific workflow generation when no test cases exist. It uses teacher-student distillation and a compact shared context, then improves SciCode results across GPT-4o, Claude Sonnet 4, and Gemini 2.5 Flash. Another study asks what small local models can do offline for Python bug detection. On 349 BugsInPy cases, LLaMA 3.2 and Mistral reach about 43% to 45% exact accuracy, with many partial answers that point to the right area but miss the full fix. The practical pattern is targeted help under clear constraints, not general autonomous coding.