Source note
Did Google's AI agents build an operating system for $916?
Coding AgentsMulti Agent Software EngineeringOpen World EvaluationSoftware Engineering AIAgent Benchmarks
Summary
The article argues that Google’s operating-system-by-agents claim is not strong evidence without the prompt, code, logs, retry count, and copying checks. It treats the demo as a useful case for independent evaluation of long-running coding agents.
Problem
- Google claimed that agents built an operating system for about $900 from a single prompt, but the public writeup leaves out details needed to judge the claim.
- The case matters because agent vendors now use long-running software tasks as proof of capability, while standard benchmarks cannot test this kind of work well.
- Missing methodology makes it hard to tell whether the result came from model capability, heavy prompt engineering, task-specific scaffolding, retries, or copied public code.
Approach
- The authors audit Google’s public blog post rather than rerunning the experiment.
- They check whether the writeup defines human intervention, reports retries and dry runs, releases artifacts, and tests for copied or memorized code.
- They separate the model from the scaffold around it: specialized roles, subagent delegation, tool access, stuck-agent restarts, and an anti-cheating component.
- They argue that open-world evaluations need stronger norms: public artifacts, clearer intervention logs, cost reporting, and independent review.
Results
- Google reported a final cost of $916.92 in API fees and a total budget of 2.6B tokens.
- Google described the task as starting from a single prompt, but the prompt later became many thousands of lines; the article says the number of prompt-writing attempts is not reported.
- Google said a few dozen subagents worked together, with specialized roles and delegation through the Antigravity 2.0 setup.
- Google said the final run needed no additional human guidance or corrections, but the article says the writeup gives 0 clear counts for manual restarts, approvals, escalations, dry runs, or retries.
- Google released 0 of the key artifacts needed for independent review: the long prompt, the generated source code, and the agent logs.
- The article reports no benchmark baseline, no similarity analysis, and no log analysis to test whether the agents copied or memorized existing toy operating-system code.