Production Controls for AI Engineering

Engineering teams testing coding agents can add two product metrics to pilot dashboards: the number of back-and-forth iterations needed to finish a request, and whether assisted code remains after review and later edits. Gemini for Google reports both measures at large scale, with a blind A/B study across 29,000 developers showing 23% fewer mean iterations per turn and about 17% higher code survival rates.

A practical version does not require a custom foundation model on day one. An internal pilot can tag agent-assisted pull requests, connect chat or IDE sessions to review outcomes, and sample code survival after a fixed window such as 30 or 60 days. Multi-language repositories also need a pre-change checklist: inspect the repository, verify APIs and tooling, check dependencies and security rules, and label checks the agent could not run. The Polyglot Protocol is useful here as a procedural template, especially because it names repository discovery and validation failures that teams already see in code review.

Teams using local coding agents need an inventory of package names, maintainers, install locations, and permissions for tools that can run shell commands. The GSD incident shows the specific failure mode: an npm package can keep its publish path even after social trust in the maintainer collapses, and agent packages may run with bash or shell permissions on a developer machine.

The immediate workflow is small enough for a security team or developer-experience team to run this week. List globally installed and project-local AI agent packages, check ~/.npm/_npx/ and .claude for leftovers, pin approved package names, and require a maintainer-control review before agents receive shell permissions. For tools installed through npm, the review should include who can publish, whether a fork changed the update path, and how developers can remove stale packages. This is supply-chain review applied to agent tooling, where an update can become a local code-execution path.

Device teams can test AI-authored microcontroller behavior by exposing a narrow driver API and running generated Lua apps inside an on-device sandbox. Resident shows a concrete pattern on ESP32: app code is pushed over Wi-Fi through a websocket, runs without compiling or flashing firmware, and can use selected capabilities such as button events and display writes while blocked from unrestricted access such as the network stack.

This fits prototypes where the interaction loop must feel local. The author cites 150 ms as the response limit for an interface to feel instant, which is hard to meet when every button press waits on a cloud model. A useful first test is a small device with a screen and buttons, such as an M5StickS3-style kit, plus two measurements Resident does not yet publish: app load latency and memory use under repeated hot loads. Safety testing should include attempts to reach blocked APIs, especially networking and hardware controls outside the exposed driver surface.