Recoleta
GitHub
Source note

Agentic AI Systems Should Be Designed as Marginal Token Allocators

Authors
Siqi Zhu
Source
arXiv
Published
Collection
Software Intelligence
Agentic AIToken AllocationCoding AgentsLLM RoutingServing Systems
Open arXivMarkdown source

Summary

The paper argues that agentic AI systems should allocate each added token by expected quality gain minus compute, latency, and risk costs. It uses a coding-agent request to show how routing, agent actions, serving, and training fit one allocation rule.

Problem

  • Flat per-token accounting treats model selection, planning, verification, KV cache use, and RL rollouts as the same kind of cost, even though they change quality, latency, and risk in different ways.
  • Local optimization can waste tokens: a cheap router choice can force extra agent verification, create serving congestion, and produce lower-quality training traces.
  • This matters for coding agents and other action-taking systems because token cuts can raise the cost of wrong commits, missed tests, stale caches, and slow service.

Approach

  • The paper defines each possible token use as an option: cheap model, frontier model, retrieval, planning, tool call, verifier, prefill, decode, KV transfer, RL rollout, reward computation, or gradient update.
  • The core rule is simple: spend the next token where task value × marginal quality gain - compute cost - latency cost - risk cost is highest.
  • It maps the same rule to 4 layers: routing as model screening, agent policy as plan-act-verify allocation, serving as prefill/decode/KV production, and training as investment in future capability.
  • It proposes shared shadow prices for compute, latency, and risk so upstream choices can account for downstream cost.
  • It turns common failures into pricing errors: over-routing, over-delegation, under-verification, serving congestion, stale rollouts, and cache misuse.

Results

  • This is a position paper and reports no empirical benchmark gains, ablations, or production measurements.
  • It gives a worked routing example: cheap model quality 0.7 at cost 1 versus frontier model quality 0.9 at cost 5; the frontier model becomes preferred when task value exceeds 20 without risk.
  • In the same example, adding wrong-action probabilities 0.05 for the cheap model and 0.01 for the frontier model with risk price 50 shifts the crossover to about 10.
  • It claims one first-order condition covers 4 system layers: demand, action, supply, and capital.
  • It identifies 6 recurring failure modes explained by mispriced marginal tokens: over-routing, over-delegation, under-verification, congestion, stale rollouts, and cache misuse.
  • It recommends concrete artifacts for evaluation and operation: regret-based router evaluation, autonomy schedules by action class, and logged shadow prices for prefill, decode, and KV resources.

Link