Recoleta
GitHub
Source note

Context-Augmented Code Generation: How Product Context Improves AI Coding Agent Decision Compliance by 49%

Authors
Drew Dillon, Kasyap Varanasi
Source
arXiv
Published
Collection
Software Intelligence
Code GenerationCoding AgentsProduct ContextRetrieval Augmented GenerationSoftware Engineering Benchmarks
Open arXivMarkdown source

Summary

The paper tests whether product context helps AI coding agents follow team decisions that are absent from source code. On an 8-task benchmark, Claude Code + Brief raises weighted decision compliance from 46% to 95%, with a workflow confound that the authors acknowledge.

Problem

  • AI coding agents can produce code that compiles while missing product decisions stored in specs, wikis, audits, and product tools.
  • This matters because missed decisions can create compliance failures, deprecated UI usage, wrong feature flags, or changes that need human rework.
  • Existing code benchmarks such as SWE-bench focus on issue resolution and do not score whether code follows team-specific product decisions.

Approach

  • The paper defines decision compliance as a weighted pass/fail score for task-specific gotchas, such as required audit logging, approved UI components, feature flags, ORM choices, and auth wrappers.
  • The benchmark uses a clean-room Next.js 14 app called Prism Analytics, with 8 realistic tasks, 41 weighted decision points, 15 seeded product decisions, 3 personas, 5 customer signals, and 3 competitor profiles.
  • The baseline is Claude Code with codebase access only, using Claude Sonnet 4.6 for code generation.
  • The augmented setup adds Brief, which retrieves recorded decisions, personas, customer signals, and competitive context during spec generation and mid-build consultation; Opus 4.6 handles planning and Sonnet 4.6 handles code generation.
  • Scoring uses regex checks over git diffs plus human review, with 3 independent runs per task per setup, for 48 total runs.

Results

  • Decision compliance improves from 19/41 points, or 46%, for Claude Code to 39/41 points, or 95%, for Claude Code + Brief, a 49 percentage point gain.
  • Claude Code + Brief reaches 100% compliance on 6/8 tasks, compared with 2/8 for Claude Code; tasks at 0% drop from 2/8 to 0/8.
  • Blocking violations fall from 5 to 0, deprecated pattern use falls from 3 to 0, and any type count falls from 9 to 0.
  • Merge-ready tasks increase from 2/8, or 25%, to 8/8, or 100%; cost per merge-ready task drops from $2.07 to $0.66, a 68% reduction, even though total cost rises from $4.13 to $5.28.
  • The augmented setup writes 838 tests while the baseline writes 0; lint and typecheck pass rates are 100% for both, while test pass rate is 0% for the baseline and 100% for the augmented setup.
  • The evidence is limited by the small clean-room benchmark, 16 PR pairs, one human reviewer, and a confound: Brief changes both available context and the coding workflow through specs, acceptance criteria, and mid-build guidance.

Link