Source note
CUJBench: Benchmarking LLM-Agent on Cross-Modal Failure Diagnosis from Browser to Backend
Summary
CUJBench is a benchmark for diagnosing software failures that start with browser-visible symptoms and extend into backend telemetry. It packages each incident as a deterministic snapshot so different LLM agents can be compared on the same cross-modal diagnosis task.
Problem
- Existing root-cause analysis benchmarks focus on backend logs, traces, and metrics, while web-agent benchmarks focus on completing tasks in working apps. Neither tests diagnosis that links a broken user journey in the browser to backend causes.
- Real incident diagnosis often starts with screenshots, network requests, or console errors, then moves to traces, logs, and recent changes. Benchmarks miss this browser-to-backend reasoning path.
- Live environments add noise and make agent comparisons unreliable, so a useful benchmark needs fixed evidence and repeatable tool outputs.
Approach
- The paper builds CUJBench, a benchmark of failed critical user journeys (CUJs) captured as frozen multi-modal snapshots with frontend evidence, backend observability, and operational context.
- It uses two open-source applications, OpenTelemetry Demo and Tractor Store, to cover backend-dominant, browser-dominant, and cross-modal failures.
- The corpus contains 87 labeled scenarios across five fault families: baseline, browser proxy faults, backend flag faults, compound faults, and frontend mutations.
- Scenario creation uses an LLM-assisted generation pipeline that produced 120 candidates, then filters them through a multi-agent review loop with two SRE reviewers, a senior reviewer, and human verification. 87 of 120 scenarios were admitted.
- Evaluation uses a fixed tool interface with deterministic cached responses, including browser tools such as screenshots and HAR/network data, backend tools such as logs and traces, and context tools such as recent changes and service topology.
Results
- On the benchmark, six frontier models reach only 19.7% overall accuracy, with a reported ceiling of 52%, so the task is far from solved.
- The paper evaluates six models under three baselines: retrieval-only, browser-only, and full-toolset.
- A main empirical claim is that browser-only agents outperform full-toolset agents in aggregate, which suggests that extra tools often push agents into broader but less focused evidence search.
- The benchmark analysis says the main bottleneck is cross-modal synthesis: agents often retrieve the key evidence but fail to connect it to the true cause.
- Prior backend-only work cited in the paper reports low perfect RCA accuracy on OpenRCA, at 3.9% to 12.5% across five frontier models, which frames CUJBench as a harder and broader diagnosis setting.
- The excerpt does not include a per-model score table, per-dataset split metrics, or exact browser-only vs full-toolset deltas.