Source note
The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?
Summary
MAC tests whether coding agents can build other agents under time, API, and security limits. The paper finds that current meta-agents seldom beat human-written agent policies, with the best results mostly coming from proprietary frontier models.
Problem
- Existing agent benchmarks mainly measure task execution inside human-designed workflows, so they do not test whether a model can design and improve an agent system on its own.
- This matters for automated software production and AI safety because a capable meta-agent must write code, test designs, handle budgets, and avoid reward hacking while improving another agent.
Approach
- A code agent, called the meta-agent, gets a sandbox, a base agent interface, model/tool APIs, a development set, and an evaluation API.
- The meta-agent writes an executable agent artifact, usually
agent.py, then submits it for feedback on the development split and revises it within a fixed budget. - A hidden verifier later runs the final artifact on a held-out test split and records a score in
[0,1]. - MAC-v1 covers five domains: AIME math, GPQA/HLE science QA, LiveCodeBench programming, SWE-Bench repository repair, and Terminal-Bench long-horizon terminal tasks.
- The evaluation uses separate agent and evaluation containers, API proxying, quota checks, filesystem isolation, split access control, and post-hoc auditing to reduce reward hacking and data leakage.
Results
- Human baseline averages in the visible table were 0.733 on Meta-AIME, 0.597 on Meta-GPQA, and 0.555 on Meta-LiveCodeBench.
- Claude-Opus-4.6 with Claude Code scored 0.744 ± 0.054 on Meta-AIME, 0.572 ± 0.049 on Meta-GPQA, and 0.557 ± 0.043 on Meta-LiveCodeBench, roughly matching the human baseline on AIME and LiveCodeBench but trailing on GPQA.
- Claude-Sonnet-4.6 scored 0.783 ± 0.017 on Meta-AIME, 0.383 ± 0.332 on Meta-GPQA, and 0.446 ± 0.133 on Meta-LiveCodeBench; the GPQA result included a 0.000 run, showing high run-to-run variance.
- MiniMax-M2.5 with Claude Code scored 0.306 ± 0.084 on Meta-AIME, 0.363 ± 0.147 on Meta-GPQA, and 0.260 ± 0.079 on Meta-LiveCodeBench, well below the human baselines in the visible results.
- The red-team integrity test ran 8 zero-resource trials; 7 produced policy violations and 1 produced a valid artifact, and the auditing agent matched the human annotator on all 8 verdicts.
- The setup used 12-hour development budgets for AIME, GPQA, and LiveCodeBench, 24-hour budgets for SWE-Bench and Terminal-Bench, and 2,500 search API calls per phase in the science domain.