Source note
ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents
Summary
ClawMark is a benchmark for AI coworker agents that work across multiple days, multiple turns, and multiple data types while the environment changes outside the agent's control. It measures a gap that earlier agent benchmarks mostly miss: staying useful when files, emails, calendars, and other tools keep changing between workdays.
Problem
- Existing agent benchmarks usually test one static session, but real coworker agents face work that continues across days.
- In real office settings, the world changes between turns: new emails arrive, calendar entries move, records get edited, and evidence may appear in PDFs, images, audio, video, or spreadsheets.
- This matters because an agent that looks strong in a frozen episode may fail when it must refresh state, track progress over time, and act on raw multimodal evidence.
Approach
- The paper introduces ClawMark, a benchmark with 100 tasks across 13 professional scenarios. Each task spans 2 to 6 turns (mean 3.6), with one turn per in-universe workday.
- Tasks run in a dynamic sandbox with five stateful services: filesystem, email, calendar, knowledge base, and spreadsheet. The environment changes between turns through announced "loud events" and unannounced "silent mutations."
- Evidence is multimodal and untranscribed. The release includes 1,072 raw artifacts such as PDFs, images, audio, video, and spreadsheets.
- Scoring is fully rule-based. The benchmark uses 1,537 deterministic Python checkers, including 55 red-line constraints, and does not use LLM-as-judge scoring.
- The authors evaluate seven frontier agent systems under a common harness to compare how current models handle long, stateful coworker workflows.
Results
- ClawMark claims a distinct benchmark setting relative to prior work: among the listed benchmarks, it is the only one with multi-day tasks = yes, dynamic environment = yes, full multimodal input = yes, and rule-based verification.
- On the 100-task benchmark, the top weighted score is 75.8 from Claude Sonnet 4.6. The next best are Claude Opus 4.6 at 74.6 and GPT-5.4 at 72.0.
- On the stricter Task Success metric, the best model is Claude Opus 4.6 at 20.0%, followed by Claude Sonnet 4.6 at 14.0% and GPT-5.4 at 9.0%. This shows that partial progress is common, but full end-to-end completion is still rare.
- Red-line failure rates vary by model: 3.6% for Claude Sonnet 4.6, 5.5% for Claude Opus 4.6, 3.6% for GPT-5.4, 14.5% for Qwen 3.6 Plus, and 9.1% for Kimi K2.5.
- Turn-level analysis on 73 three-turn tasks shows that 6 of 7 models drop on Day 2 after the first external environment update. The reported gap between Claude Sonnet 4.6 and GPT-5.4 narrows from +6.5 points on Day 1 to +4.0 points on Day 3.
- The main empirical claim is that current frontier agents can often make partial progress in dynamic office workflows, but they still struggle to fully complete multi-day tasks once the outside world changes.