Source note
Learned or Memorized ? Quantifying Memorization Advantage in Code LLMs
Code LLM EvaluationMemorization DetectionBenchmark ContaminationPerturbation AnalysisSoftware Engineering Benchmarks
Summary
This paper measures how much code LLM performance depends on memorized benchmark data instead of general code understanding. It uses input perturbations to estimate a model's "memorization advantage" across many software engineering tasks and benchmarks.
Problem
- Code LLM training data is usually opaque, so benchmark leakage is hard to verify directly.
- If evaluation examples appeared in training, reported benchmark scores can overstate real generalization.
- This matters for code generation, repair, testing, and security tasks, where a model that recalls known examples may fail on small variations of new problems.
Approach
- The paper defines memorization advantage as the performance gap between an original input and slightly perturbed versions of that input.
- For each sample, it creates progressively stronger but still human-preserving perturbations, prompts the model on each variant, and measures task performance against the reference output.
- It uses the maximum drop between consecutive perturbation levels as the sample's sensitivity score: high sensitivity suggests reliance on memorized patterns or weak generalization.
- The study evaluates 8 open-source code LLMs on 19 benchmarks spanning code generation, code understanding, vulnerability detection, bug identification, test generation, and program repair.
- It compares sensitivity distributions across models and across benchmarks, using Mann-Whitney U tests with Bonferroni correction, 3 repeats, temperature 0.3, and top_k 0.5.
Results
- StarCoder shows high sensitivity on some benchmarks, reaching 0.8 on APPS, while QwenCoder stays below 0.4 on most benchmarks.
- Code summarization has low sensitivity, usually <0.3, which the paper reads as stronger generalization on that task.
- Test generation has higher sensitivity, around 0.4-0.7 with p < 0.001, making it one of the hardest settings for stable generalization.
- Defects4J has lower sensitivity than other program repair benchmarks, around 0.2-0.4 with p < 0.01, versus 0.5-0.8 on other repair datasets.
- CVEFixes stays below 0.1 across models, which weakens the common claim that this benchmark is heavily contaminated.
- The paper's main claim is that suspected leakage benchmarks such as CVEFixes and Defects4J may reflect real generalization more than simple memorization, while sensitivity varies widely by model family and task category.