Context
PR #44 closes #43 by hooking api_request_error and persisting model-unavailable alerts. The unit suite + a headless PoC against an isolated HERMES_HOME cover the plugin code end-to-end (see the PR #44 smoke-test comment).
What we could not exercise in the sandbox is the real HTTP path: hermes cron tick calling out to the provider, getting a 404, and agent/conversation_loop.py invoking the api_request_error hook for real.
Why it blocks
When HERMES_HOME points to a fresh tmp dir, the sandbox doesn't carry Nous Portal credentials. The cron run dies with:
RuntimeError: Hermes is not logged into Nous Portal.
Run `hermes model` to re-authenticate.
before any HTTP request leaves the box, so the hook is never reached via the real call site. Same blocker applies to any provider that requires interactive auth (portal flow, OAuth device codes, etc.).
A related minor friction: hermes gateway restart refuses to rewrite the systemd unit when HERMES_HOME resolves to a tmp dir (safety check), so the systemd-managed gateway keeps serving the developer's real ~/.hermes. Sessions launched from the sandbox shell still load the plugin from the sandbox correctly — but anything that goes through the gateway daemon (e.g. unattended cron under systemd) does not.
Options for a follow-up
- Local mock provider — point a cron job at a
base_url we control (e.g. a tiny aiohttp server in tools/) that always returns 404 Not Found. Hermes calls it, the hook fires for real, no credentials needed. Easiest to wire into CI later.
- Non-interactive auth bootstrap — a script that copies/refreshes a portal token into the sandbox
HERMES_HOME so hermes cron tick can call Nous without manual hermes model. Closest to the real production code path, but token handling is per-developer.
- Document the limitation — accept that the headless PoC is the gate, and add a one-paragraph note to
ONBOARDING.md § Test Isolation Contract explaining why the live-gateway smoke test isn't run on every PR. Cheapest.
Suggested next step
Option (1). The aiohttp mock provider is ~30 lines, lives in tools/, and gives us a deterministic 404 source we can also reuse for the planned 410 / persistent-5xx widening mentioned in PR #44 Out of scope.
Acceptance criteria
- A
tools/mock_404_provider.py (or similar) that serves 404 Not Found on POST /chat/completions.
- A
tests/smoke_e2e_cron.sh (or pytest equivalent) that: spins up the mock, points a sandbox cron job at it, runs hermes cron tick, asserts a row in model_unavailable_alerts and the alert string in telemetry.log.
- README / ONBOARDING note on how to run the smoke test locally.
Context
PR #44 closes #43 by hooking
api_request_errorand persisting model-unavailable alerts. The unit suite + a headless PoC against an isolatedHERMES_HOMEcover the plugin code end-to-end (see the PR #44 smoke-test comment).What we could not exercise in the sandbox is the real HTTP path:
hermes cron tickcalling out to the provider, getting a 404, andagent/conversation_loop.pyinvoking theapi_request_errorhook for real.Why it blocks
When
HERMES_HOMEpoints to a fresh tmp dir, the sandbox doesn't carry Nous Portal credentials. The cron run dies with:before any HTTP request leaves the box, so the hook is never reached via the real call site. Same blocker applies to any provider that requires interactive auth (portal flow, OAuth device codes, etc.).
A related minor friction:
hermes gateway restartrefuses to rewrite the systemd unit whenHERMES_HOMEresolves to a tmp dir (safety check), so the systemd-managed gateway keeps serving the developer's real~/.hermes. Sessions launched from the sandbox shell still load the plugin from the sandbox correctly — but anything that goes through the gateway daemon (e.g. unattended cron under systemd) does not.Options for a follow-up
base_urlwe control (e.g. a tiny aiohttp server intools/) that always returns404 Not Found. Hermes calls it, the hook fires for real, no credentials needed. Easiest to wire into CI later.HERMES_HOMEsohermes cron tickcan call Nous without manualhermes model. Closest to the real production code path, but token handling is per-developer.ONBOARDING.md § Test Isolation Contractexplaining why the live-gateway smoke test isn't run on every PR. Cheapest.Suggested next step
Option (1). The aiohttp mock provider is ~30 lines, lives in
tools/, and gives us a deterministic 404 source we can also reuse for the planned 410 / persistent-5xx widening mentioned in PR #44 Out of scope.Acceptance criteria
tools/mock_404_provider.py(or similar) that serves404 Not FoundonPOST /chat/completions.tests/smoke_e2e_cron.sh(or pytest equivalent) that: spins up the mock, points a sandbox cron job at it, runshermes cron tick, asserts a row inmodel_unavailable_alertsand the alert string intelemetry.log.