Source note
An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code
Local LlmsBug DetectionPython CodeCode IntelligenceOffline Inference
Summary
This paper tests whether small open-weight LLMs running fully offline can find real Python bugs. On 349 BugsInPy cases, LLaMA 3.2 and Mistral reached about 43% to 45% accuracy, with many additional responses that spotted the problem area but missed the exact fix.
Problem
- The paper studies bug detection with locally deployed LLMs, where cloud models are often unusable because of privacy, cost, or internet requirements.
- Prior work focused more on cloud systems, synthetic tasks, or narrower bug classes, so the practical value of offline consumer-hardware models for real Python bugs was unclear.
- This matters for code intelligence and developer tooling because teams may want bug analysis on proprietary code without sending source code to external services.
Approach
- The authors evaluate two local models, LLaMA 3.2 8B and Mistral 7B, using Ollama on a MacBook Pro 14-inch (2021) with M1 Pro and 16GB RAM, fully offline with 4-bit quantization.
- They use the BugsInPy benchmark and successfully process 349 of 501 bugs from 17 Python projects after excluding cases where source retrieval or function extraction failed.
- For each bug, they extract the enclosing function around the buggy line and give the model a zero-shot prompt: “Here is a Python function. It contains a bug. Find the bug and explain how to fix it.”
- They score free-text answers with an automated keyword method based on tokens added in the fix, labeling outputs as correct, partial, or wrong; a manual check of 50 responses found high agreement with the automatic labels.
- They also break results down by project and by 9 bug types, such as Null/None checks, return value bugs, indexing, and complex multi-component bugs.
Results
- Overall accuracy was 43.3% for LLaMA 3.2 (151/349 correct, 171 partial, 27 wrong) and 44.4% for Mistral (155/349 correct, 161 partial, 33 wrong).
- The gap between models was small: McNemar’s test p = 0.68, with paired outcomes of 126 both-correct, 169 both-wrong, 25 LLaMA-only correct, and 29 Mistral-only correct. The paper treats this as no significant difference.
- Accuracy varied a lot by project: examples include PySnooper 100% for both models, black 73.7% / 68.4%, fastapi 69.2% / 61.5%, pandas 38.5% / 44.8%, and tqdm 0.0% / 14.3% for LLaMA/Mistral.
- By bug type, both models did best on Null/None Check bugs (59.5% LLaMA, 60.8% Mistral) and Return Value bugs (51.3% for both). They did poorly on Type Conversion (0.0% for both) and Other/Complex bugs (21.7% LLaMA, 16.7% Mistral).
- Inference was feasible on consumer hardware: average response time was about 7 seconds for LLaMA 3.2 and 13 seconds for Mistral, for total runs of about 40 minutes and 75 minutes.
- The strongest practical claim is that local models can catch a meaningful share of real bugs and often narrow the search area, but precise localization remains weak when the bug depends on wider program context or cross-function behavior.