Source note
Generalizing Test Cases for Comprehensive Test Scenario Coverage
Summary
TestGeneralizer turns one developer-written test into more tests that cover the intended behavioral scenarios of a method. It targets scenario coverage rather than branch coverage and reports better coverage than EvoSuite, gpt-o4-mini, and ChatTester on a Java benchmark.
Problem
- Existing automated test generation tools mostly optimize code coverage, but developers write tests to check requirement-driven scenarios that may not match control-flow branches.
- Important scenarios are often missing when the first test is written and get added later after bugs or issue reports, which makes testing slower and less complete.
- Requirements are often implicit in code and tests, so the task is to infer the hidden test pattern and generate valid scenario variants from a single initial test.
Approach
- The system takes a focal method, one existing test, and the project codebase, then runs a 3-stage pipeline called TestGeneralizer.
- Stage 1 uses Masked Oracle Modeling (MOM): it mutates the original test's assertions into executable wrong alternatives and asks the LLM to pick the correct one. If the model is unsure or wrong, it retrieves project facts through program analysis to improve understanding.
- Stage 2 asks the LLM to write a test scenario template: a compact plan with variation points such as input style, object type, or API choice. It then instantiates those variation points into concrete scenario instances with primary and alternative oracles.
- To make variation-point detection more accurate, the prompts are automatically tuned. To make instantiated scenarios project-valid, the system retrieves code facts with tools such as CodeQL and JDTLS.
- Stage 3 generates executable tests for each scenario instance and iteratively repairs compile errors, runtime errors, and assertion failures until the test passes or the iteration limit is reached.
Results
- Evaluation covers 12 open-source Java projects, 506 multi-test focal methods, and 1,637 test scenarios.
- Against EvoSuite, TestGeneralizer improves mutation-based scenario coverage by 57.67% and LLM-assessed scenario coverage by 59.62%.
- Against gpt-o4-mini, it improves mutation-based scenario coverage by 37.44% and LLM-assessed scenario coverage by 32.82%.
- Against ChatTester, it improves mutation-based scenario coverage by 31.66% and LLM-assessed scenario coverage by 23.08%.
- In a field study, the authors submitted 27 generated tests that developers had missed; 16 were accepted and merged into the official repositories.
- The paper also claims the method works consistently across both commercial and open-source LLMs, including ChatGPT and DeepSeek-V3.1.