Source note
Agora: Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents
Summary
Agora is a three-agent LLM system for finding protocol-level logic bugs in consensus implementations. It reports 15 previously unknown safety bugs across Raft, EPaxos, HotStuff, and BullShark, while the tested ReAct-style LLM baselines found 0 protocol-level logic bugs.
Problem
- Consensus code can lose safety through state-dependent bugs that span elections, recovery, message order, dependency tracking, signatures, or persistence.
- Standard LLM code agents tend to find local implementation bugs, such as memory or simple logic errors, and miss bugs that need protocol-level reasoning.
- This matters because consensus failures can corrupt data in distributed storage and databases or cause financial loss in blockchain systems.
Approach
- Agora uses three LLM agents: an Orchestrator that tracks global state and prior findings, a Strategy agent that creates protocol-specific attack scenarios, and a TestGen agent that writes and runs unit tests.
- The method follows hypothesis-driven testing: each bug hypothesis includes trigger conditions, an action sequence, expected faulty behavior, and oracle checks.
- A small knowledge library gives the agents consensus bug patterns and constraints for CFT and BFT protocols, so the search avoids unrealistic scenarios such as Byzantine assumptions in CFT systems.
- The Strategy agent varies node behavior, crashes, recovery, joins, message ordering, and conflict relationships; the TestGen agent iterates on tests until they run and expose a violation or the retry limit is reached.
Results
- Across four consensus implementations, Agora found 15 previously unknown protocol-level logic bugs: 1 in Raft, 9 in EPaxos, 4 in HotStuff, and 1 in BullShark.
- The ReAct-style baselines using GPT-5.2, Gemini 3.0 Pro Preview, Claude Sonnet 4.5, and Qwen3 Coder 480B A35B found 22 implementation bugs in total and 0 protocol-level logic bugs.
- With Agora, GPT-5.2 found 8 logic bugs, Gemini 3.0 Pro Preview found 11, Claude Sonnet 4.5 found 6, and Qwen3 Coder 480B A35B found 9; the combined unique total was 15.
- Ablation results in the excerpt show Agora found 11 bugs in that study setting; removing bug-exploitation cut this to 3, removing constraints-analyzer cut it to 1, and removing state-analyzer, scenario-generator, or reflection-loop cut it to 0.
- The paper reports that removing one component reduced effectiveness by 73% to 100%, based on the ablation table.