Recoleta
GitHub
Source note

Show HN: HermesBench – workflow reliability evals for personal AI agents

Authors
verkyyi26
Source
Hacker News
Published
Collection
Software Intelligence
Agent EvaluationPersonal AI AgentsWorkflow ReliabilityCode Agent RunnerHuman AI Interaction
Open originalMarkdown source

Summary

HermesBench evaluates full personal-agent configurations across real workflow recipes and publishes trace-backed scores. The current public baseline is 78.2 across 27 recipes, so the main value is transparent reliability testing for agent deployments.

Problem

  • Personal-agent quality depends on prompts, model/provider choice, tools, memory, safety behavior, delegation, latency, and stability, so model-only tests miss many user-visible failures.
  • Personal agents can cause harm when they finish the wrong task, use unsafe side effects, hide uncertainty, or become too slow for normal use.
  • Early agent benchmarks often lack inspectable traces and clear limits, which makes setup comparisons hard to trust.

Approach

  • HermesBench runs scenario recipes against a complete Hermes configuration, including prompt, model/provider, tools, AgentSkills, memory, gateway behavior, delegation, safety, latency, and stability.
  • Each recipe has scenario definitions, public score axes, driver closure decisions, deterministic checks, and redacted trace timelines.
  • Users can run a default single-recipe path through Codex, Claude, or another coding agent by loading the HermesBench skill and saving artifacts.
  • Full bundle runs are optional because they cost more and take longer.
  • Scoring rewards useful completion, truthful uncertainty, safe behavior, stability, prompt responses, and clear communication; lopsided scores receive penalties.

Results

  • The published baseline score is 78.2 across 27 personal-agent recipes.
  • The bundled catalog covers 10 workflow areas: context, calendar, web, reports, communication, location, travel, finance, safety, and power-user integrations.
  • The public site has 3 evidence tabs for recipes, profiles, and traces.
  • The excerpt reports 1 early public baseline and says this is not a base-model leaderboard.
  • No per-axis scores, confidence intervals, failure counts, or comparisons against other agent benchmarks are provided in the excerpt.

Link