Skip to content

v0.8.2-alpha

This alpha release packages the LLM usage and USD cost accounting work merged after v0.8.1-alpha.

Use it if you need per-job token totals, estimated or provider-reported LLM costs, or a database-backed audit trail for internal OpenAI SDK calls, Codex CLI runs, and Kilo CLI runs.

Highlights

  • Loreley now writes LLM usage to a durable llm_usage_events ledger. Events are attributed by job, run token, phase, source, provider, model, API surface, token breakdown, cost source, pricing version, and raw provider summary.
  • Internal OpenAI SDK calls record usage for code embeddings, worker commit subject summaries, and trajectory summaries after the API response succeeds. Embedding usage is recorded only after response validation, so malformed responses rejected by the embedding layer do not create accepted usage rows.
  • Codex CLI usage tracking reads token_count events from codex exec --json while still using --output-last-message for the final Markdown contract. Setting LLM_USAGE_TRACKING_ENABLED=false disables the JSON usage path for built-in Codex planning and coding backends.
  • Kilo CLI runs are linked with a loreley:<job_id>:<run_token>:<phase> title. Loreley can read Kilo local usage records, preserve provider-reported costs, or estimate token-only Kilo records from the configured pricing table.
  • The UI API exposes read-only usage routes, and the Streamlit Overview and job detail views show usage totals by source, phase, and model.

Upgrade Notes

Database migration required

This release bumps INSTANCE_SCHEMA_VERSION from 12 to 13. Take a Postgres backup before upgrading a non-disposable database, then run:

uv run loreley db migrate
uv run loreley db validate

When DB_AUTO_MIGRATE=true, API, scheduler, and worker startup can apply the same migration automatically. When DB_AUTO_MIGRATE=false, run uv run loreley db migrate before starting those processes.

The migration creates the llm_usage_events table and supporting indexes. It does not backfill historical usage from older job artifacts or CLI sessions.

Usage tracking and pricing

Usage tracking is enabled by default:

LLM_USAGE_TRACKING_ENABLED=true

Disable it only when an environment cannot support the extra CLI usage mode, or when you do not want usage rows stored:

LLM_USAGE_TRACKING_ENABLED=false

Cost values use two sources:

  • provider_reported: provider or CLI returned a USD cost. Loreley stores that value directly.
  • estimated: provider returned tokens only, and Loreley matched the event to a local pricing rule.

Set either pricing input when the built-in OpenAI defaults are not enough:

LLM_USAGE_PRICING_PATH=/path/to/pricing.json
LLM_USAGE_PRICING_JSON='{"version":"local-2026-05","prices":[...]}'

Loreley does not fetch pricing at runtime. Store a versioned local price table so historical costs do not silently change when provider prices change.

Kilo usage source

Kilo provider-reported usage is best-effort and depends on Kilo's local state. Set WORKER_KILOCODE_USAGE_DB_PATH when the default Kilo DB location is not available to the worker process:

WORKER_KILOCODE_USAGE_DB_PATH=/path/to/kilo.sqlite

If Loreley cannot read Kilo usage after a successful agent invocation, the job still succeeds and the usage event is marked unavailable.

Read-only API routes

New usage routes are served under /api/v1:

  • GET /usage/summary
  • GET /usage/events/page
  • GET /jobs/{job_id}/usage

These routes are read-only and use the same UI API process as the rest of the dashboard.

Fixes

  • Retry-attempt usage is carried through retryable agent exceptions, coercion failures, post-check retries, and exhausted retry paths, so billable attempts are not lost when a later attempt succeeds or fails differently.
  • Planning and coding artifacts now include an aggregate usage_summary next to structured usage_events.
  • Kilo token-only records bill tokens.input and tokens.cache.read as separate counters. OpenAI/Codex-shaped usage still treats cached input as a subset of input tokens for pricing.
  • The usage implementation was refactored to keep the Cremona structural-debt gate clean without updating the baseline.

Validation

  • Local release validation: 708 passed, 4 skipped.
  • Focused checks:
  • uv run pytest tests/core/map_elites/test_code_embedding.py -q
  • uv run pytest tests/core/worker/test_agent_backends.py -q
  • uv run pytest tests/core/test_llm_usage.py -q
  • git diff --check.
  • Documentation build: uv run --with mkdocs-material mkdocs build --strict.
  • GitHub CI tests, tests, and cremona / cremona passed on PR #39 head 03a763a38c3876673ae5b04d9a36991eee1d2d36.
  • Cloud Codex review threads were resolved and Codex reacted +1 to the final PR head.
  • Add a durable LLM usage ledger with token and USD cost attribution for internal OpenAI SDK calls, Codex CLI, and Kilo CLI.
  • Add read-only usage APIs and dashboard usage summaries.
  • Add schema version 13 and configurable local pricing for estimated costs.