Recoleta
GitHub
Source note

Neurodata Without Boredom: Benchmarking Agentic AI for Data Reuse

Authors
Ling-Qi Zhang, Kristin Branson
Source
arXiv
Published
Collection
Software Intelligence
Agentic AICode IntelligenceScientific Data ReuseNeuroscience DataHuman AI Interaction
Open arXivMarkdown source

Summary

The paper tests whether Claude Code and Codex can turn messy neuroscience datasets into a shared format for neural decoding. The main finding is that agents often solve single conversion steps, but full error-free dataset reuse still needs human review.

Problem

  • Neuroscience datasets are split across labs, file formats, APIs, and experiment designs, so reusing them can take substantial manual work.
  • Standards such as NWB make files easier to load, but field meanings and analysis choices often remain unclear.
  • This matters for cross-dataset analyses and brain-behavior foundation models, where data preparation can become the bottleneck.

Approach

  • The benchmark uses 8 recent mouse neural population recording papers with shared data and code, including NWB files, consortium APIs such as IBL and Allen Brain Observatory, Python files, and MATLAB files.
  • Each agent received the paper, methods text, released code, and raw data, then wrote convert_data.py to produce converted_data.pkl in a prescribed subject/session/trial format.
  • The target task was fixed across datasets: train a linear decoder from neural activity to task or behavioral variables.
  • The evaluation used outcome metrics, such as dataset statistics and decoder balanced accuracy, plus manual process ratings for data loading, trial construction, neural preprocessing, variable construction, missing-data handling, and code efficiency.
  • The study ran Claude Code Opus 4.6 and Codex GPT 5.4 three times per dataset, for 48 agent runs.

Results

  • All 48 runs produced converted data in the required format and yielded decoder-performance values.
  • On supervised datasets with human references, agents passed many outcome checks, such as Allen2P format checks at 11/12 for Claude Code and 9/12 for Codex, and Lee2025 checks at 15/15 for both agents.
  • Full end-to-end success was rare: Allen2P had 0/3 successful Claude Code runs and 0/3 successful Codex runs by the table's end-to-end criterion, while Lee2025 reached 3/3 for both agents.
  • Manual process scores, measured as the proportion of subtasks rated at least ok, ranged from 0.813 to 0.938 for Claude Code across datasets and from 0.885 to 1.000 for Codex.
  • The paper reports 169 incorrect or concerning trial-subtask cases per agent for error analysis, with many errors tied to filtering choices, time resolution, processing decisions, assumptions from variable names, and ambiguous semantics.
  • Agents-as-judges were unreliable, especially without ground-truth references, so the authors argue for interactive human review in scientific data reuse.

Link