Three tests call the live API outside the HELM_LIVE guard; suite needs a key to go green

[MrJev]

Hi — I reviewed Helm for mrjev.com (an independent directory of Jev projects) and ran the suite on a clean clone. Two things worth your time, both in the tests rather than the product. I made no live Jev calls; everything below is against a local stand-in on TYPESAFE_ENDPOINT.

Tested at 2cdedbadcd87695a5d566abdedfb6c6653ec61c1, python:3.12-slim, python -m unittest discover -s tests.

1. Three tests bill the user's account, outside the HELM_LIVE guard

test_jev_live.py is explicit that "most runs of this suite should not need the network and must not cost anyone money", and gates itself on HELM_LIVE=1. But test_supervise.TestRealDispatch calls the real S.supervise(), which dispatches and then asks Jev to judge. With a key configured and HELM_LIVE unset:

$ python -m unittest tests.test_supervise.TestRealDispatch
Ran 3 tests in 3.051s
OK

$ wc -l < wire.jsonl        # POSTs that reached my stand-in
3
$ # each one:
['awaiting_input', 'failure_severity', 'needs_human', 'status', 'task_satisfied']

Three supervision requests, on a plain discover run. Small money, but it is the exact thing the live file exists to prevent.

2. The suite cannot go green on a clean clone

Without a key, 7 tests fail (failures=6, errors=1). supervise() checks keystore.require_api_key() before the card lookup — deliberately, and the comment explains why — so the four TestPreconditions cases get the no-key message instead of the condition they assert:

AssertionError: 'unknown agent' not found in "No TypeSafe API key configured -- ..."

Setting any key string fixes 6 of the 7, still fully offline:

	no key	TYPESAFE_API_KEY=apikey_dummy, offline
result	failures=6, errors=1, skipped=18	failures=1, skipped=18

The last one, test_a_hanging_agent_is_killed_at_the_timeout, needs a reachable Jev: the kill works, but the verdict comes back error rather than timeout because the judge could not be asked.

Both would be covered by moving TestRealDispatch behind skip_unless_live, or by pointing it at a stub endpoint. Happy to send a PR if you'd like one.

Unrelated, and much smaller

A 200 whose body carries no answers ({"answers":{}}) is not treated as a failure. compose() defaults every signal to its midpoint, agent_name is None, and the run reports:

ESCALATE -- no installed agent fits this task
  signals: kind None | confidence 0.0 | blast 1.5 | clarity 1.5 | breadth 1.5 ...
  judged by: jev
  not auto-executing because:
    - selected agent 'None' is not in the routable set

exit 0. It fails closed, which is the important half. But it is reported as a routing conclusion "judged by: jev" rather than as the decision layer failing, and exit 4 is what the README documents for that. HTTP 500, a non-JSON body and a refused connection all correctly gave exit 4.

---

For what it's worth, the thing I came to check held up. I gave a machine exactly two agents and a stand-in that answered cursor-agent at confidence 0.99 anyway:

criteria actually offered to Jev:  ["aider", "codex", "none"]

ESCALATE -- no installed agent fits this task
  not auto-executing because:
    - selected agent 'cursor-agent' is not in the routable set

Never on the menu, and caught again behind that. Your comment calls that branch "unreachable by construction" — it is, and it still does its job when something upstream lies.