Source note
Test-Oriented Programming: rethinking coding for the GenAI era
Summary
The paper proposes Test-Oriented Programming (TOP), a software development style where developers review generated tests and let the LLM generate production code from those tests. The authors built a proof-of-concept tool, Onion, and show that this workflow can complete a small command-line program, with clear limits and failure modes.
Problem
- Current LLM coding tools still leave developers inspecting production code, so the abstraction level stays close to ordinary programming.
- Natural-language specifications are ambiguous, and LLM outputs are non-deterministic, so direct delegation of implementation is hard to trust.
- The paper asks whether developers can shift their review effort to test code instead of production code, which matters if GenAI is going to reduce manual coding work in a reliable way.
Approach
- TOP uses natural-language specifications to generate test code first, then uses those tests to generate production code.
- Developers review and edit the generated YAML configuration, system structure, and test files, but do not directly edit production code in the intended workflow.
- The authors implemented this idea in Onion, a command-line Python tool that iteratively generates project structure, acceptance tests, class tests, and then implementation code until tests pass or the run aborts.
- The evaluation used one small BibTeX command-line application with features to add entries, list entries, and search text, and compared two models: GPT-4o-mini and Gemini 2.5-Flash.
Results
- Onion was run 5 times per model on the same task, for 10 total runs.
- All 10 runs succeeded in producing the target application.
- In 0 runs did the developers need to directly modify production code.
- In 2 runs total (1 with GPT-4o-mini, 1 with Gemini 2.5-Flash), developers had to add comments to the test code to guide generation after repeated failures on the same tests.
- For GPT-4o-mini, the authors report 1 run where they had to modify the test code before checking the implementation.
- The paper does not report standard software metrics such as pass rate by benchmark, time saved, token cost, accuracy, or comparison against Copilot, TDD, or multi-agent baselines. The strongest concrete claims are that production code could be generated from reviewed tests on a small task, outputs varied across runs, and Gemini 2.5-Flash tended to generate longer, more commented code than GPT-4o-mini.