Do Machines Struggle Where Humans Do? LLM and Human Comprehension of Obfuscated Code

Summary

This paper tests whether LLMs find obfuscated code hard in the same places humans do. Reasoning-tuned models track human task difficulty, while coder and instruction-tuned models show weak alignment.

Problem

Code obfuscation preserves behavior while changing names and control flow, so it can test whether a model follows program logic or relies on surface cues.
Standard code benchmarks can hide brittle code understanding because familiar identifiers, idioms, and control-flow shape may carry much of the answer.
This matters for code intelligence and software security: an agent that misreads obfuscated code may give wrong answers in auditing, reverse engineering, malware analysis, or protected-code maintenance.

Approach

The study reuses Nguyen et al.'s human dataset: 50 undergraduate programmers, 20 function-level output-prediction tasks, Python and JavaScript, and five obfuscation tiers from L0 to L3.
It adds Dataset B with 250 snippets from HumanEval-X, CruxEval-X, and LeetCode, each instantiated across the same five tiers.
It evaluates Llama, Qwen, DeepSeek, Phi, SmolLM, CodeLlama, and DeepSeek-Coder variants on exact output prediction.
Obfuscation tiers include identifier renaming, adversarial renaming, control-flow flattening, and combined renaming plus control-flow flattening.
The analysis uses Schulte's Block Model to locate failures at atom, block, relational, and macro levels, with metrics for accuracy, chain-of-thought length, dispatcher complexity, and high-confidence wrong answers.

Results

On Dataset A, DS-R1-Qwen-7B is the strongest model: 63.8% accuracy at L0, 64.2% at L1, 51.8% at L1b, 57.0% at L2, and 56.2% at L3.
Human accuracy across all participants drops from 40.46% at L0 to 31.09% at L3. Beginners drop from 34.21% to 23.08%, and intermediates drop from 45.10% to 33.33%.
Reasoning-tuned models align with human task difficulty: Spearman correlations are Qwen3-0.6B ρ=0.30, Phi-4 Mini ρ=0.47, SmolLM3-3B ρ=0.36, and DS-R1-Qwen-7B ρ=0.37, all with p≤0.003.
Coder and instruction-tuned models show near-zero human alignment: Llama-3.1-8B ρ=0.08, CodeLlama-7B ρ=0.02, and DeepSeek-Coder-6.7B ρ=0.10.
Control-flow flattening hurts accuracy as dispatcher complexity rises: Python while-if state count correlates with accuracy at r=-0.196 with q=3.08×10^-23; JavaScript dispatch-call references have r=-0.130 with q=2.76×10^-7.
Low token limits strongly reduce strong reasoning-model performance, with OR=12.5, 95% CI [6.4, 24.2], p<0.001 for the reported comparison.

Link

https://arxiv.org/abs/2606.31725v1