Source note
Neural Change Prediction: Relating Software Changes to Their Effects and Vice Versa
Summary
Neural Change Prediction trains models on synthetic code and configuration mutations paired with observed output changes, then uses those pairs to predict either the effect of a change or the code change needed for a desired effect. The paper reports high accuracy on CSS and Python tasks, with fine-tuned GPT-4.1 well above reported LLM baselines.
Problem
- Developers often need to know which code change will cause a desired behavior change, or what behavior a proposed code change will cause.
- Static reasoning and current LLMs struggle to infer dynamic program behavior without execution; the paper reports only 10% to 33% accuracy for current LLM systems on these tasks.
- Better predictions could aid debugging, feature localization, change impact analysis, software evolution, and repair, while proposed fixes can still be checked by running tests.
Approach
- For a given program and test inputs, the system applies many synthetic mutations to source code or configuration files.
- It runs the original and mutated program, then records the input, original output, mutated output, changed code, and mutation location as training data.
- It trains models in both directions: code change plus current output -> output change, and desired output change -> likely code location or exact code edit.
- CSS experiments use natural-language intents such as changing a rendered element's color; Python experiments use output changes for given inputs.
- The study fine-tunes GPT-4.1/GPT-4.1-mini and also considers GPT-oss, CodeLlama, Qwen, and simpler machine learning baselines.
Results
- CSS: fine-tuned GPT-4.1 reached up to 95% accuracy for predicting correct CSS changes under general learning across templates.
- CSS: project-specific learning on one template reached 100% accuracy.
- Python desired-behavior task: fine-tuned GPT-4.1 reached 82.6% accuracy for correct change location with single mutations.
- Python desired-behavior task: fine-tuned GPT-4.1 reached 71.6% accuracy for the exact change with single mutations.
- Python effect prediction: fine-tuned GPT-4.1 reached 95% accuracy for predicting output change from a single code mutation, and 99% accuracy for multiple mutations.
- Baseline context: current LLM systems are reported at 10% to 33% accuracy on these tasks.