Recoleta
GitHub
Source note

AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering

Authors
Rajesh Kumar, Waqar Ali, Junaid Ahmed, Najma Imtiaz Ali, Shaban Usman
Source
arXiv
Published
Collection
Software Intelligence
Multi Agent LLMAutonomous Software EngineeringExecution GroundingSwe BenchCode Repair
Open arXivMarkdown source

Summary

AgentForge is a multi-agent software engineering system that requires every code change to pass real execution in a Docker sandbox before the change moves forward. The paper argues that this execution-grounded loop gives stronger correctness feedback than plain code generation.

Problem

  • LLMs can write plausible code, but they often cannot tell whether the code actually works in a real repository.
  • Real bug fixing needs a loop over existing code, tests, execution output, and revision; one-shot generation misses multi-file context and regression checks.
  • Prior agent systems may simulate execution or make verification optional, which lets wrong assumptions pass through the pipeline.

Approach

  • AgentForge splits the work across five agents: Planner, Coder, Tester, Debugger, and Critic.
  • The key rule is mandatory execution grounding: every generated patch runs inside a network-isolated Docker sandbox before it can be accepted or revised further.
  • The system retrieves two kinds of context for each task: similar past solved tasks from episodic memory and relevant files from a live repository index.
  • The Coder edits files with minimal unified diffs when possible, the Tester writes pytest cases, and the Debugger uses real stdout/stderr and test failures for up to 3 repair attempts.
  • The paper also formulates the workflow as an MDP over repository states, where reward is 1 only when all fail-to-pass tests pass and no pass-to-pass tests regress.

Results

  • On SWE-bench Lite, AgentForge reaches 40.0% resolution.
  • The paper says this beats single-agent baselines by 26 to 28 percentage points.
  • Evaluation uses SWE-bench Lite, a benchmark of 300 real GitHub issues from 11 Python repositories.
  • The execution environment is a constrained Docker sandbox with 512 MB RAM, 0.5 CPU, no network access, and a 64-process PID cap.
  • The debugger loop retries failed code up to 3 times.
  • The excerpt does not include the full ablation table, but the paper claims ablations show that both execution feedback and role decomposition improve performance on their own.

Link