Recoleta
GitHub
Source note

Articulate but Wrong: Self-Review Failures in LLM-Based Code Modernization

Authors
Gokul Chandra Purnachandra Reddy, Aditya Lolla, Harsha Sanku
Source
arXiv
Published
Collection
Software Intelligence
LLM Code ModernizationSelf ReviewBehavioral EquivalenceCode IntelligenceSoftware Testing
Open arXivMarkdown source

Summary

The paper finds that LLMs often change Python 2 behavior during Python 3 modernization, and same-model self-review misses many of those failures. Its main practical claim is that production modernization pipelines need external behavioral checks instead of plain self-approval.

Problem

  • LLM agents can produce code that runs and looks plausible while changing the legacy program's observable behavior.
  • This matters because compile checks and basic tests may pass after migration, leaving downstream callers with different values or types.
  • The paper tests whether the model that wrote the migrated code can catch its own behavior-changing output.

Approach

  • The authors build a 60-snippet Python 2 corpus: 20 semantic traps, 20 syntactic traps, and 20 benign controls that need no real modernization.
  • They run 1,980 modernization calls across 11 production LLMs, 7 model families, and 3 prompt phrasings at temperature 0.
  • A type-strict behavioral oracle compares each candidate against the Python 2 contract, including value and return type, so int-to-float drift is counted.
  • After generation, the same model reviews the legacy snippet and its candidate output and answers whether behavior was preserved.

Results

  • Semantic traps drift in 39.7% of attempts versus 7.0% on benign controls and 12.7% on syntactic traps, across 660 calls per category.
  • Numeric-semantics traps are the main failure class, with 57% drift; lazy-evaluation traps drift at 21%, while type-model and literal-syntax traps stay near baseline.
  • Models agree on which snippets are hard: mean pairwise Pearson r=0.52 across 55 model pairs, with a core of numeric snippets failing for at least 8 of 11 models under every prompt.
  • Same-model self-review endorses 83 of 262 semantic drift cases, or 31.7%; for numeric drift, it misses 75 of 207 cases, or 36%.
  • Per-model self-miss rates range from 0% on five models to 100% on one model, and semantic drift rates range from 5.6% to 46.7% in the abstract and up to 65.0% in the per-model table.
  • Method choices change measured rates: permissive equality lowers semantic drift from 39.7% to about 26.7%, while a fence-only parser raises benign-control drift from 7.0% to about 23%.

Link