MUCOCO: Automated Consistency Testing of Code LLMs

Summary

MuCoCo is an automated test method for finding consistency failures in code LLMs. It mutates coding tasks into semantically equivalent versions and checks whether the model changes its behavior across the pair.

Problem

Code LLMs can give different answers for programs that mean the same thing, but common coding benchmarks mostly test correctness, not consistency.
Hand-built benchmarks are static, take manual effort, and can leak into newer models over time.
Consistency matters because unstable behavior can break trust in code generation, execution prediction, and other software tasks.

Approach

MuCoCo takes a coding query, creates semantically equivalent mutants, runs the LLM on the original and mutated versions, and compares the outcomes.
It uses 11 semantic-preserving mutations in three groups: lexical, syntactic, and logical. Examples include variable renaming, loop rewrites, DeMorgan rewrites, and constant unfolding.
A correctness oracle checks whether each output is correct, incorrect, or invalid using expected outputs and test suites.
A consistency oracle flags an error when the original and mutated queries lead to different outcomes, such as pass vs fail, valid vs invalid, or different failed tests.
The evaluation covers 4 benchmarks, 4 task types, and 7 code LLMs.

Results

Across all settings, MuCoCo found inconsistencies in 14.82% of generated test pairs: 21,924 / 147,935 cases.
Average model accuracy on these tasks was 69.13% (102,092 / 147,680), so inconsistency remained common even when models were often correct.
By model, inconsistency rates ranged from 2.92% for GPT-5 to 21.22% for LLaMA-3.1; GPT-4o reached 18.15%, Gemma-3 17.69%, DeepSeek-V3.2 16.53%, Codestral 14.51%, and Qwen2.5 12.73%.
By mutation type, lexical mutations exposed the most inconsistencies overall at 16.28% (12,974 / 79,707), followed by logical at 13.91% (6,259 / 45,008) and syntactic at 11.59% (2,691 / 23,220).
By task, inconsistency was 27.15% for code generation, 22.35% for MCQ, 18.99% for output prediction, and 6.65% for input prediction.
Against Turbulence on the Turbulence dataset, MuCoCo reports 41.39% error rate versus 33.8% for Turbulence overall, with 1,781,288 / 4,302,713 errors versus 279,241 / 826,084; the paper states MuCoCo is up to 6x more effective in some settings and finds new classes of consistency errors.

Link

http://arxiv.org/abs/2604.19086v1