Source note
SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications
Summary
SmartEval is a benchmark for judging Solidity contracts generated by LLMs from natural-language specifications. It pairs 9,000 generated contracts with expert implementations and scores them with a five-metric rubric focused on specification fidelity and state-machine behavior.
Problem
- LLMs can write Solidity, but smart contracts need exact business logic, state transitions, access control, and security checks because deployed bugs can lock or lose funds.
- Existing evaluation often checks syntax or compilation, which misses whether a contract follows the natural-language requirements and FSM specification.
- The paper targets a measurement gap: researchers need a repeatable way to compare LLM-generated smart contracts against expert implementations.
Approach
- SmartEval uses the FSM-SCG dataset, which contains 21,976 specifications with natural language, FSMs, and expert Solidity code; the main release evaluates 9,000 generated contracts.
- A multi-agent pipeline parses each specification into a structured schema, generates Solidity, audits the result, refines code when medium-or-higher issues appear, and scores the final contract.
- The score combines five metrics: Functional Completeness 25%, Variable Fidelity 15%, State Machine Correctness 15%, Business Logic Fidelity 35%, and Code Quality 10%.
- Composite scores are recomputed from raw metric values, which avoids accepting aggregate scores produced by an LLM evaluator.
- Validation uses three checks: human expert scoring, Slither static analysis, and a five-condition ablation study with 300 contracts per condition.
Results
- On 9,000 generated contracts, SmartEval reports an average composite score of 81.54, standard deviation 12.87, and compilation success rate 86.54% across 8,824 checked contracts.
- The generated contracts score 81.54 versus 73.25 for expert ground truth, a +8.29 point gain or +11.3% under the paper's rubric.
- Per-metric generated scores are 84.45 Functional Completeness, 84.62 Variable Fidelity, 83.12 State Machine Correctness, 76.73 Business Logic Fidelity, and 83.85 Code Quality; Business Logic Fidelity is the weakest metric.
- Grade distribution is 7.3% A, 66.4% B, 23.1% C, and 3.2% D/F, with complete failures reported at 2.2%.
- The main failure modes among 2,398 lower-performing contracts are logic omissions at 35.3% and state transition errors at 23.4%; high-complexity specs with 8+ functions and 5+ states average 71.8 versus 87.2 for low-complexity specs.
- Validation reports human expert agreement within 0.34 points, Slither agreement with the LLM auditor at 79.4%, and a gate ablation where disabling refinement raises standard deviation from 4.89 to 10.31 and reduces compilation by 5.2 percentage points.