Source note
Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction
Coding Agent BenchmarkCode IntelligenceAutomated Software ProductionMulti Domain EvaluationContamination Resistance
Summary
Tencent WorkBuddy Bench is an openly released benchmark for coding agents across repository engineering, web development, office workflows, and security tasks. It aims to measure realistic work while reducing searchable-prompt contamination through task rewriting, versioning, and reproducible evaluation.
Problem
- Existing public benchmarks often reuse web-searchable issue or repository content and focus mainly on single-issue software fixes, making scores vulnerable to memorization.
- Production-derived benchmarks better reflect real usage but are commonly closed, limiting auditability and independent reproduction.
- Agents increasingly work across code, web, office, and security artifacts, so a broader evaluation suite matters; however, these domains require different verification methods.
Approach
- Construct each task by reverse-engineering a real commit, pull request, CVE, or business scenario, then rewriting it as a colloquial, role-played request that withholds the root cause, target files, and reference solution.
- Match task categories, roles, modes, and difficulty to aggregate internal usage distributions without reusing raw user prompts or sessions.
- Package tasks in a Harbor-style directory with an agent-visible workspace separated from post-episode evaluation assets; release prompts, environments, tests, harnesses, and reference solutions openly.
- Evaluate four tracks with domain-specific verifiers: hidden tests for Code, rule and LLM/VLM/agent judges for Web, deterministic checks plus evidence-grounded LLM judging for Office, and deterministic scoring for Security.
- Use dataset versioning and optional canary strings to manage post-release exposure; the construction prevents searchable recovery of task prompts but does not guarantee contamination-free evaluation.
Results
- The initial release contains 260 tasks: 80 Code, 70 Web, 50 Office, and 60 Security. Scores are reported separately because the tracks use different metrics, and the suite reports no overall average.
- The Code subset includes 34 tasks anchored to real upstream commits, 24 clean-room or ported tasks, and 22 synthetic workspaces; only 10 of 80 tasks are bug fixes.
- Code task admission requires baseline verifier reward <= 0.3 and oracle reward = 1.0, ensuring the untouched workspace does not already satisfy the contract and a reference patch can reach full reward.
- Code tasks span five requester roles and are concentrated at higher difficulty levels: 42 of 80 are editorially hard and 40 of 80 are L4 on the repository-complexity ladder.
- The excerpt describes a cross-model leaderboard evaluated with CodeBuddy Code and Claude Code, but it does not provide quantitative model scores, dataset-level baselines, or comparative performance results.