CASCADE: Detecting Inconsistencies between Code and Documentation with Automatic Test Generation

Summary

Cascade detects mismatches between method documentation and code by turning documentation into executable tests, then checking those tests against both the original code and a code version regenerated from the same documentation. The paper targets high precision, with the main guard against false alarms coming from this two-step execution check.

Problem

The paper solves code-documentation inconsistency detection at the method level, where docs say one thing and the implementation does another.
This matters because wrong API documentation misleads users, can cause downstream bugs, and adds maintenance cost.
A practical detector must keep false positives low, since developers still need to inspect each report by hand.

Approach

Cascade uses an LLM to read natural-language documentation and generate unit tests that encode the documented behavior.
It runs those tests on the existing implementation. If all tests pass, Cascade reports no evidence of inconsistency.
If some generated tests fail, Cascade does a second check: it asks the LLM to generate a fresh implementation from the same documentation.
Cascade reports an inconsistency only when at least one test changes from fail on original code to pass on regenerated code (f2p > 0), and no test changes from pass on original code to fail on regenerated code (p2f = 0).
The tool also includes a repair loop for generated tests that do not compile, with up to 3 repair attempts.

Results

Evaluation uses a new dataset with 71 inconsistent and 814 consistent code-documentation pairs from open-source Java projects.
The paper claims this dataset is the first one built from real, developer-confirmed semantic inconsistencies in executable Java projects.
On additional Java, C#, and Rust repositories, Cascade found 13 previously unknown inconsistencies.
Of those 13 findings, 10 were later fixed by developers.
The excerpt does not provide standard quantitative metrics such as precision, recall, F1, or accuracy, and it does not include baseline comparison numbers.

Link

http://arxiv.org/abs/2604.19400v1