Source note
The Productivity-Reliability Paradox: Specification-Driven Governance for AI-Augmented Software Development
AI Assisted DevelopmentCode IntelligenceSoftware ReliabilitySpecification Driven DevelopmentHuman AI Interaction
Summary
The paper defines the Productivity-Reliability Paradox: AI coding tools can speed up small tasks while adding review load, churn, and stability risk in real software systems. It proposes specification-driven governance, where teams use explicit specs and executable tests to constrain AI-generated code.
Problem
- AI coding studies report gains of 20–56% on scoped tasks, while mature-codebase evidence shows slowdowns and reliability loss.
- Faster code generation can shift work into review, testing, security checks, and rework, so team delivery metrics may stay flat.
- The issue matters because AI coding is already common: the paper cites 84% of professional developers using or planning to use AI tools and 46% of code output coming from AI suggestions in instrumented environments.
Approach
- The authors review 67 sources published from January 2022 to April 2026: 29 peer-reviewed studies, 18 preprints, 12 industry reports, and 8 grey-literature sources.
- They define the Productivity-Reliability Paradox using three moderators: task abstraction level, codebase maturity, and developer experience.
- They identify two amplifiers: code review bottlenecks and context window limits. In simple terms, AI can create more code than teams can review, and models may miss project context that sits outside their prompt window.
- They propose the Specification Governance Model: teams write specs and tests before delegating code generation, then use those artifacts to limit what the agent builds and to check its output.
- They compare GitHub Spec Kit and the Test-Driven AI Agent Definition pipeline as concrete examples, then add a four-month pilot across three industry teams.
Results
- Positive productivity evidence includes Peng et al. 2023: 95 developers completed an HTTP-server task 55.8% faster with Copilot.
- GitHub’s 2024 RCT with 202 developers reports a 53.2% higher test-pass likelihood and quality gains in readability (+3.62%), reliability (+2.94%), maintainability (+2.47%), and conciseness (+4.16%).
- Google’s enterprise RCT reports about 4,867 participants, a 26% pooled throughput increase, and a 21% speedup at Google.
- Counterevidence includes the METR RCT: 16 experienced open-source developers working on 246 tasks were 19% slower with AI tools, even though they predicted a 24% speedup.
- The paper cites DORA 2024: a 25% increase in AI adoption is associated with a 7.2% drop in delivery stability and a 1.5% drop in throughput.
- The specification-driven evidence is thinner: TDAD reports mutation scores of 86–100% across four domains, and the paper describes a four-month pilot across three teams, but the excerpt does not provide detailed pilot metrics.