Source note
RAT: RunAnyThing via Fully Automated Environment Configuration
Summary
RAT is a language-agnostic agent framework that configures runnable environments for arbitrary code repositories. It targets a core failure point in repository-level code agents and reports higher setup success than prior methods on a new 2,000+ repository benchmark.
Problem
- Repository-level software agents often fail before they can test or run code because the project environment is missing, broken, or unclear.
- Existing setup methods depend on curated artifacts such as Dockerfiles, CI logs, or language-specific rules, which limits coverage on messy real-world repositories.
- This matters because execution is required for verification, benchmark creation, training data synthesis, reinforcement from runtime feedback, and reliable deployment.
Approach
- RAT builds environments with a multi-stage agent pipeline: detect the repository’s main language, inspect project files and docs, pick a starting container image, then iteratively install and verify dependencies inside a sandbox.
- Its ImageRetriever module uses repository semantics plus Docker Hub search to choose a better base image than a fixed default such as
python:3.10oropenjdk:17. - The agent runs with either a fixed standard plan or an automated plan that writes progress into
plan.mdas external memory for longer configuration sessions. - RAT adds task-specific tools for reading and editing files, parsing CI configs, switching language versions, retrieving similar past issues, recovering from errors, and generating smoke tests when a repo has no tests.
- The paper also introduces RATBench, an exec-verified benchmark of 2,000+ GitHub repositories across Python, Java, Rust, and JavaScript/TypeScript, sampled to cover different project sizes, popularity levels, and artifact availability.
Results
- On RATBench, RAT reports ESSR of 63.2% on Python, 41.3% on Java, 98.7% on Rust, and 68.7% on JS/TS.
- Against SWE-agent with the same backbone (DeepSeek-V3), RAT improves ESSR by 47.7 points on Python (63.2 vs 15.5), 12.0 points on Java (41.3 vs 29.3), 42.0 points on Rust (98.7 vs 56.7), and 16.9 points on JS/TS (68.7 vs 51.8). The paper states an average gain of 29.6% over strong baselines.
- On Python-specific baselines, RAT beats Repo2Run by 18.4 points (63.2 vs 44.8), pipreqs by 27.4 points (63.2 vs 35.8), Zero-shot by 48.0 points (63.2 vs 15.2), and Installamatic by 56.5 points (63.2 vs 6.7).
- Across Python evaluation settings, RAT reports 50.5% ESSR in S1 (artifact-guided), 69.5% in S2 (artifact-free), and 92.0% in S3 (test-deficient with synthesized checks), with token use and latency of 451.3K / 41.6 min, 455.2K / 59.4 min, and 122.2K / 14.4 min.
- Ablations on Python show the base system at 63.2% ESSR, versus 40.5% without ImageRetriever, 55.7% without the specialized toolset, and 40.5% with automated planning only. Full RAT uses 421.9K tokens and 24.3 min on average in this setting.