Source note
Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Summary
PDB is a debugging benchmark that checks whether a model makes targeted fixes or rewrites large parts of the program. The paper shows that strong coding models often pass tests while making many unnecessary edits, so unit-test accuracy alone hides weak debugging behavior.
Problem
- Current debugging benchmarks score models by unit-test pass rate, so a minimal fix, a full rewrite, and some forms of hard-coding can look equally good.
- Real debugging needs fault localization and small, reviewable edits; broad regeneration is costly and risky in real codebases.
- Existing evaluation also misses partial progress on multi-bug programs, because fixing one bug and fixing none can receive the same binary score.
Approach
- The authors introduce Precise Debugging Benchmarking (PDB), a pipeline that turns existing coding datasets into debugging benchmarks by injecting verified bugs into correct programs.
- PDB first creates atomic bugs with known ground-truth edits, then composes independent bugs into multi-bug programs so each bug can be measured separately.
- The benchmark adds two metrics beyond unit tests: edit-level precision for how many edits were actually needed, and bug-level recall for how many bugs were truly fixed.
- It includes two released benchmarks: PDB-Single-Hard with 5,751 hard examples built from single-line bugs, and PDB-Multi with 256 examples built from contiguous multi-line bugs.
- The evaluation also tests single-shot, iterative, and agentic debugging setups to see whether extra attempts or feedback improve precise editing.
Results
- On PDB-Single-Hard, GPT-5.1-Codex gets 76.1% unit-test score but only 39.7% precision and 71.7% recall. DeepSeek-V3.2-Thinking gets 79.0% unit score with 45.0% precision and 71.2% recall.
- Models with lower test-pass rates can be more precise debuggers: Qwen3-Coder-480B gets 70.3% unit score but 65.8% precision and 77.2% recall.
- The best precision on PDB-Single-Hard comes from Claude-4.5-Sonnet at 71.8% precision, 81.4% recall, 75.7% unit score, and Gemini-2.5-Pro at 71.4% precision, 83.5% recall, 78.1% unit score. Even these models stay below 72% precision.
- On PDB-Multi, the same gap remains: GPT-5.1-Codex has the top unit-test score at 77.0% but only 27.9% precision and 59.4% recall; Claude-4.5-Sonnet reaches 65.9% precision, 73.9% recall, 64.8% unit score; Gemini-2.5-Pro gets 57.8% precision, 73.2% recall, 72.7% unit score.
- Iterative and agentic debugging improve unit-test score and recall, but the paper says they do not improve precision in a meaningful way; even Claude-Code reaches only about 50% precision in the agentic setting.
- Prompting matters: freeform debugging reduces precision and recall. The paper reports that even strong models drop below 60% precision under freeform prompts, and Gemini-2.5-Pro loses about 40 percentage points of precision in that ablation.