LLM coding research is demanding harder artifacts and tighter ownership

ClassEval-Pro targets a gap between isolated function synthesis and repository repair: writing a complete Python class with shared state, method dependencies, and domain logic. The benchmark has 300 tasks across 11 domains, built partly from GitHub repositories created after January 1, 2025. Its tasks are larger and more connected than the older ClassEval set.

The results show that multi-method coordination is still hard. Across five large language models, holistic generation reaches only 27.9% to 45.6% class-level Pass@1. Bottom-up generation helps weaker models by up to 9.4 percentage points, while compositional generation can fall to 1.3%. In 500 manually labeled failures, logic errors account for 56.2% and dependency errors for 38.0%, making cross-method coordination the main measured failure mode.

Comet-H treats research software as a workspace where theory, code, benchmarks, public claims, evidence, and open obligations must stay aligned. Its controller rereads the repository, scores 17 prompt families, and forces grounding plus audit when papers or READMEs change. The reported portfolio includes 46 research-software repositories; one static-analysis tool reaches F1 = 0.768 on a 90-case benchmark, compared with 0.364 for the next-best baseline.

EvoRec applies the same control problem to service recommendation. It updates service facts with model editing and constrains decoding with finite automata and tries, so generated service names stay valid and non-duplicate. The paper reports a 25.9% relative Recall@5 gain over baselines and a 22.3% gain over fine-tuning in evolving-service settings, although the summary evidence does not include dataset names or absolute scores.

Hot Fixing in the Wild studies urgent GitHub repairs across more than 61,000 repositories in the Hao-Li/AIDev dataset. The authors combine local large language model classification with timing filters: the pull request must open within 12 hours of issue creation and close within 24 hours of PR creation. This matters because urgency language alone produced many false candidates.

The measured hot fixes are smaller and less reviewed than routine fixes. With Qwen labels, hot-fix pull requests average 2.7 commits, 3.9 files, 25.7 added lines, and 9.3 deleted lines. Routine fixes average 4.9 commits, 27.7 files, 90 added lines, and 54.4 deleted lines. Test edits appear in 29.73% of Qwen-labeled hot fixes, compared with 54.42% of routine fixes. Merge rates are higher for hot fixes, including similar rates for bot and human authors under the Qwen-labeled subset.

Two position papers focus on the human side of AI-assisted software work. Cognitive Atrophy and Systemic Collapse argues that teams can accumulate “epistemological debt,” meaning maintainers know less about what a system does than the code execution implies. Its evidence is mostly conceptual and case-based. It cites Amazon Q Developer migrations of 30,000 production applications to Java 17, a reported 79% acceptance rate for generated code reviews without manual modification, and two claimed 2026 incidents linked to GenAI-assisted changes.

The curriculum paper makes a related education claim. It says computer science programs should require architecture, verification, deployment, monitoring, security, cost control, and ownership of AI-enabled systems. It gives no new benchmark, but its strongest practical recommendation is clear: students should test behavior above the API layer because large language model components can change behavior while interfaces remain stable.