Skip to content

v0.7.8-alpha

This alpha release packages the changes landed after v0.7.7-alpha.

Use it if you need safer recovery for stuck jobs, better visibility into lease health, and explicit repair commands when scheduler and worker runs go wrong.

Highlights

  • Automatic recovery for RUNNING jobs with stale or malformed lease state. The scheduler now requeues reclaimable jobs until they exhaust a recovery budget, then marks them FAILED.
  • New job repair CLI. Use uv run loreley jobs inspect, uv run loreley jobs ls, and uv run loreley jobs retry to inspect lease state and requeue stuck jobs.
  • Better lease visibility in uv run loreley status. The status payload now includes a job_leases section with total RUNNING, stale, malformed, and recovery-exhausted counts, plus the current lease timing settings.
  • Safer worker fencing. Worker success, failure, and candidate-publication writes now use run_token-scoped lease ownership checks so stale workers cannot overwrite a newer attempt.
  • Shared repo lock coordination. When the scheduler and worker share WORKER_REPO_WORKTREE, their base-repo mutation paths use the same cross-process lock.

Upgrade Notes

Database schema reset required

Loreley still does not ship migrations. This release adds lease-recovery columns and indexes to evolution_jobs and bumps the instance schema version.

If you are upgrading an existing development database created before this change, reset it before running status, scheduler, or worker:

uv run loreley reset-db --yes

If you skip the reset, commands may fail with database errors such as a missing lease_expires_at column.

New configuration knobs

Review these settings when you tune recovery behavior:

  • SCHEDULER_STALE_RUNNING_RECLAIM_BATCH_SIZE
  • SCHEDULER_STALE_RUNNING_MAX_RECOVERY_ATTEMPTS
  • WORKER_JOB_LEASE_TTL_SECONDS
  • WORKER_JOB_HEARTBEAT_INTERVAL_SECONDS

See Configuration for details.

New operational commands

Use these commands when you operate the new lease-recovery flow:

  • uv run loreley jobs ls --failed-stale --limit 20
  • uv run loreley jobs inspect <job-id>
  • uv run loreley jobs retry <job-id>
  • uv run loreley jobs retry --failed-stale --limit 10
  • uv run loreley status --json

See Managing jobs, Job lease recovery, and Status for full examples.

Validation

  • CI continues to run uv run pytest.
  • Local validation for this release candidate: 361 passed in 17.22s.
  • Automatic recovery for stale or malformed worker leases, with a bounded retry budget.
  • New loreley jobs inspection and retry commands plus richer status lease reporting.
  • Safer shared-repository coordination and run_token fencing so stale workers cannot overwrite newer attempts.