Recoleta
GitHub
Source note

Can Coding Agents Reproduce Findings in Computational Materials Science?

Authors
Ziyang Huang, Yi Cao, Ali K. Shargh, Jing Luo, Ruidong Mei, Mohd Zaki, et al.
Source
arXiv
Published
Collection
Software Intelligence
Coding AgentsScientific ReproducibilityMaterials ScienceAgent BenchmarkCode Intelligence
Open arXivMarkdown source

Summary

AutoMat tests whether coding agents can reproduce claim-level results from computational materials science papers. The main finding is negative: the best tested agent succeeds on 54.1% of 85 claims, and paper-only workflow recovery is the hardest case.

Problem

  • Coding agents that pass software engineering benchmarks may fail in computational science because papers omit procedure details, require domain tools, and need scientific judgment.
  • The problem matters because an agent can run code and produce plausible files while using the wrong method or evidence.
  • Computational materials science is a hard test case because workflows can involve DFT, molecular dynamics, ML models, custom code, HPC jobs, and post-processing.

Approach

  • The authors build AutoMat with 85 claims curated by materials science subject matter experts from real papers.
  • Each task gives an agent a claim, paper, metadata, and optional artifacts; hidden expert reproduction steps are kept for evaluation.
  • Tasks cover from-paper reproduction, from-artifact reproduction, and from-artifact interpretation.
  • Five agent settings are evaluated: an AutoMat-specific orchestrated agent using Claude Sonnet 4.6, Claude Code with Claude Opus 4.6, Claude Code with Claude Sonnet 4.6, Claude Code with Kimi K2.5, and Codex CLI with GPT-5.4.
  • A separate evaluator agent inspects traces, logs, files, and reports, then scores runs on a 1-5 scale; success means a score of 4 or 5.

Results

  • Claude Code with Opus 4.6 is best overall, with a mean reproducibility score of 3.52 and a 54.1% success rate across 85 claims.
  • Codex CLI with GPT-5.4 is weakest overall, with a mean score of 2.44 and a 23.5% success rate.
  • From-paper reproduction has mean scores of 1.5 to 2.2 and near-zero success rates across systems.
  • From-artifact reproduction is easier, with mean scores of 3.1 to 4.1 and success rates of 39% to 77%.
  • From-artifact interpretation remains uneven, with success rates of 33% to 50%.
  • The LLM evaluator is calibrated on 40 SME-scored runs, reaching quadratic-weighted kappa 0.69 and within-1 score accuracy 0.80.

Link