Problem
The cloud-run balance preflight (PR #TBD, arenabench/arenabench/balance.py) queries GET https://openrouter.ai/api/v1/credits with the OPENROUTER_API_KEY from the operator's shell. The trials themselves authenticate with the key exported from SSM (/arenabench/openrouter_api_key, see arenabench/arenabench/cloud.py::available_credentials). When those are different accounts, the preflight measures the wrong balance in either direction: it can wave through a run whose SSM account is empty, or refuse a run whose SSM account is funded.
Today the two keys are the same account in practice, and a wrong answer degrades to exactly the pre-preflight behavior (submit and lose the run) or to one --skip-balance-check, so this is a correctness gap in the check's coverage, not a regression.
Constraints discovered
available_credentials deliberately reads names only (WithDecryption=False): the secret must not transit the operator's machine. Decrypting the SSM key locally to run the credits call is therefore not an acceptable fix.- The preflight must keep failing OPEN (a broken check must not block a funded run) — see the module doc of
balance.py.
Options a fix could weigh
- Run the credits call from inside the substrate (a tiny preflight Batch job or Lambda holding the SSM key) and refuse from there before the fan-out.
- Store a non-secret account fingerprint beside the SSM key (e.g. the key's OpenRouter label or a hash) and compare it against the local key's, warning on mismatch.
- At minimum: print which key source the check used, so the transcript records the assumption.
Done
Either the preflight demonstrably measures the account the trials will spend from, or the transcript states explicitly that it measured the local key and that the SSM key may differ. Tests in arenabench/tests/test_balance.py / test_cloud.py pinned accordingly.
Problem
The cloud-run balance preflight (PR #TBD,
arenabench/arenabench/balance.py) queriesGET https://openrouter.ai/api/v1/creditswith theOPENROUTER_API_KEYfrom the operator's shell. The trials themselves authenticate with the key exported from SSM (/arenabench/openrouter_api_key, seearenabench/arenabench/cloud.py::available_credentials). When those are different accounts, the preflight measures the wrong balance in either direction: it can wave through a run whose SSM account is empty, or refuse a run whose SSM account is funded.Today the two keys are the same account in practice, and a wrong answer degrades to exactly the pre-preflight behavior (submit and lose the run) or to one
--skip-balance-check, so this is a correctness gap in the check's coverage, not a regression.Constraints discovered
available_credentialsdeliberately reads names only (WithDecryption=False): the secret must not transit the operator's machine. Decrypting the SSM key locally to run the credits call is therefore not an acceptable fix.balance.py.Options a fix could weigh
Done
Either the preflight demonstrably measures the account the trials will spend from, or the transcript states explicitly that it measured the local key and that the SSM key may differ. Tests in
arenabench/tests/test_balance.py/test_cloud.pypinned accordingly.