Agent Execution Boundaries

Platform engineers running LLM gateways can add a budget decision before every provider request in an agent run. The practical shape is a gateway hook, sidecar, or SDK middleware that estimates worst-case cost from current input tokens, the effective output cap, and a versioned price table, then reserves that amount atomically across scopes such as run, user, and key. After the call, it commits actual usage and releases the unused reserve.

This targets the cost pattern that ordinary monthly or per-key budgets miss: an agent loop that resends accumulated context and can cross 50K input tokens by step 20. The agent also needs machine-readable budget state. Headers and RFC 9457 problem-detail errors can tell the agent when to choose a cheaper model, cut context, or stop cleanly. A useful first test is to replay real agent traces with parallel branches, missing price metadata, and growing context, then measure added latency, blocked calls, and dollars reserved versus dollars spent.

Teams adding RAG and tool use to support workflows can build the identity path as a required part of every request. The retriever should attach access metadata to chunks, authenticate the user, filter retrieved chunks through fine-grained authorization, and send only authorized text to the model. Tool calls need the same treatment through MCP with OAuth 2.1 or through ordinary APIs with tokens and gateway controls.

Agent workflows need separate records for the human user, the agent actor, the delegated scope, and the action taken. A support desk flow can split work across limited sub-agents, issue short-lived signed JWTs with delegation claims, and log the actor, delegator, human user, role, and scope for each operation. A small validation set should include restricted documents, expired credentials, and a delegated agent trying to call a tool outside its scope.

Developers evaluating local inference on consumer GPUs can add a dedicated benchmark lane for Kortex on Windows 11 before selecting a runtime for oversized GGUF models. The relevant test is narrow: a machine with a 20 GB GPU, enough RAM, one or two NVMe drives, and a 70B quantized model that exceeds VRAM. Compare Kortex with llama.cpp partial offload on the same prompt, context size, and decoding settings, then record tokens per second, output match, drive layout, and GPU residency plan.

Kortex reports Llama-3.3-70B Q4_K_M at 1.95 tokens per second on a Radeon RX 7900 XT 20 GB system, compared with 0.21 tokens per second for llama.cpp b9860 Vulkan with 30 of 80 layers offloaded on the same hardware. The current adoption boundary is clear: the streaming path is Windows-only, there is no HTTP server or multi-turn REPL, and Linux streaming has not been tested. That points to an evaluation and batch-inference path first, with service integration left for a later build.