Recoleta
GitHub
Source note

CODEFUSE-DEBENCH: An Empirical Study on Readability, Recompilability, and Functionality

Authors
Puzhuo Liu, Yuhan Huang, Jianlei Chi, Peng Di, Yu Jiang
Source
arXiv
Published
Collection
Software Intelligence
Binary DecompilationCode IntelligenceLLM EvaluationAutomated RepairSoftware Security
Open arXivMarkdown source

Summary

CODEFUSE-DeBench evaluates binary decompilers by asking whether their output is readable, can be recompiled after bounded repair, and behaves like the original binary. The study finds a large gap between code that compiles and code that preserves behavior.

Problem

  • Binary decompilers are often judged with surface metrics such as line counts, character matches, or readability scores, which do not show whether the recovered code can be reused.
  • This matters for reverse engineering, security analysis, automated patching, cross-architecture migration, and static analysis on binaries when source code is missing or unreliable.
  • Readable decompiled code can still be wrong, while messy code can preserve behavior, so one metric gives users poor tool guidance.

Approach

  • DeBench uses 240 atomic test functions across 8 source-code dimensions, compiled into 640 binaries across compilers, optimization levels, debug-symbol settings, architectures, and related build choices.
  • It evaluates 5 whole-program decompilers: IDA, Ghidra, RetDec, Angr, and BinaryAI.
  • Readability is scored with URAF, an LLM-judged rubric with 18 sub-dimensions covering naming, structure, types, semantic clarity, and context.
  • Recompilability is measured by iterative compile-and-repair with 3 independent repair LLMs under a fixed 50-iteration budget.
  • Functionality is checked with Frida-based differential dynamic tracing at program, function, and instruction levels.

Results

  • The best decompiler plus repair-LLM pair reaches 22.3% Exact+Partial program-level behavioral overlap, but only 1.2% exact stdout match.
  • The paper reports an approximately 50 percentage-point drop from recompilability to functional equivalence, showing that successful recompilation often fails to preserve runtime behavior.
  • Optimization and compiler choice change outcomes: O3 gives the lowest readability but the highest functionality, and Clang produces 2.6x higher functionality than GCC while scoring lower on readability.
  • Decompiler choice matters more than repair-LLM choice: the cross-decompiler functional swing is 20x, while the cross-LLM swing is 1.6x.
  • Failure analysis finds syntactic noise that repair can often fix, type-system collapse at about 19% of all repair errors, and upstream losses such as ARM64 relocation idioms and C++ ABI features that repair LLMs do not recover.

Link