Recoleta
GitHub
Source note

FLARE: Fine-Grained Diagnostic Feedback for LLM Code Refinement

Authors
Yinsheng Yao, Hongxiang Zhang, Weixi Tong, Tianyi Zhang
Source
arXiv
Published
Collection
Software Intelligence
Code IntelligenceLLM Code RepairFault LocalizationIterative RefinementAutomated Software Production
Open arXivMarkdown source

Summary

FLARE improves iterative LLM code repair by adding line-level bug localization to execution feedback. It uses a small diagnostic model and top-k candidate search to guide refinements toward likely faulty lines.

Problem

  • LLMs often generate code that fails tests, and iterative repair is needed for code generation systems that users can trust.
  • Test failures usually describe the failing behavior without naming the faulty line.
  • Self-critiques and natural-language debugging can give broad advice that does not tell the model where to edit.

Approach

  • FLARE trains a 6-layer, 8-head bidirectional transformer diagnostic model on 10,504 Collu-Bench buggy code solutions with ground-truth bug locations.
  • It aligns generator LLM subword probabilities to code lexical units such as identifiers, operators, and keywords.
  • The diagnostic model combines lexical-unit text embeddings, syntactic token types, and aligned token probabilities, then scores each lexical unit for suspiciousness.
  • It converts lexical scores to line scores with max pooling and sends the ranked suspicious line, the current program, and execution feedback to the refinement prompt.
  • At each iteration, it searches over the top-k suspicious lines, generates one candidate revision per line, runs public tests, and keeps a passing candidate or the candidate that passes the most tests.

Results

  • On LiveCodeBench, FLARE with k=10 reaches Pass@1 scores of 46.86, 44.57, 33.71, 76.57, and 65.14 across the five base LLMs; the corresponding LLM-alone scores are 30.86, 26.86, 20.57, 46.86, and 44.00.
  • On BigCodeBench, FLARE with k=10 reaches Pass@1 scores of 64.47, 55.79, 45.61, 72.81, and 69.65; NL-Debugging scores 55.00, 47.02, 40.35, 58.33, and 56.67.
  • With k=1, FLARE beats NL-Debugging in 9 of 10 settings. The visible table shows gains over NL-Debugging from 0.18 to 7.42 percentage points in those 9 settings.
  • Candidate search adds 8.50 percentage points on average over k=1. The reported gains are 4.57 to 12.00 points on LiveCodeBench and 5.08 to 13.42 points on BigCodeBench.
  • The diagnostic model alone reaches 67% Top-1, 75% Top-3, and 89% Top-10 localization accuracy on 100 LiveCodeBench tasks, compared with LLMAO at 63%, 74%, and 87%.
  • On GPT-4o-mini, full FLARE with k=10 reaches 65.14% Pass@1 on LiveCodeBench and 69.65% on BigCodeBench; execution-only feedback reaches 58.29% and 59.47%, while diagnostic-only feedback reaches 55.43% and 58.42%. Runtime is 73.76 seconds/task for k=1 and 610.19 seconds/task for k=10 on LiveCodeBench.

Link