diff --git a/CSCS-PROPOSAL.md b/CSCS-PROPOSAL.md index 972566e..b293638 100644 --- a/CSCS-PROPOSAL.md +++ b/CSCS-PROPOSAL.md @@ -2,6 +2,16 @@ > **Status: participation confirmed.** Event on 3 November 2026, OAT ETH Zürich (rooms S15/S16, 14th floor). This document and the rest of the repo are what we're bringing to build on the day — see `prompts/` for the build order and `README.md` → Status for logistics. +## Update — 3 November 2026: first run on production Alps + +During the hackathon the full FirecREST v2 path — token, system discovery, +filesystem, job submission, polling and output — was exercised against +production Alps (`api.svc.cscs.ch/hpc/firecrest/v2`, FirecREST 2.6.1, system +`daint`): job `5245627` ran on `nid006554` and exited `0`. Findings and the +resulting client changes (mandatory Slurm account, `/metadata` 404 fallback) are +in [`docs/hot-cache-v2.md` §8](./docs/hot-cache-v2.md); the Alps guardrail +profile is `config/limits.alps.yaml` + `templates/alps/`. + ## One sentence Let a researcher submit, monitor and diagnose HPC jobs through natural language — from a chat app, not a terminal — with the whole stack running on CSCS's own open-model inference service, not a third-party API. diff --git a/README.md b/README.md index 7210c2a..2d4c34b 100644 --- a/README.md +++ b/README.md @@ -196,6 +196,7 @@ human approval gates for `sudo` and public exposure, and verifies every phase. | 06 | Multica orchestration | done and verified live | [`MULTICA.md`](./docs/MULTICA.md) | | 07 | Telegram channel | not started — requires a human to provide a BotFather token and bind the workspace channel | — | | 08 | Failure scenario | partial delivery — recorded directly against FirecREST; it still needs a Hermes re-run | [`failure-scenario-transcript.md`](./docs/failure-scenario-transcript.md) | +| — | **Production Alps (FirecREST 2.6.1, `daint`)** | auth → submit → poll → output verified live on 2026-11-03 via direct API calls; MCP path on Alps not yet run | [`hot-cache-v2.md` §8](./docs/hot-cache-v2.md#8-production-alps--first-live-validation-2026-11-03) | The MetaMCP endpoint currently aggregates six tools: five from `firecrest-mcp` and `docmind__query_docs` from the local RAG corpus. Resume from diff --git a/config/limits.alps.yaml b/config/limits.alps.yaml new file mode 100644 index 0000000..fd4d595 --- /dev/null +++ b/config/limits.alps.yaml @@ -0,0 +1,19 @@ +# Limits for production Alps (CSCS), system daint, through FirecREST v2. +# Select with FCAGENT_LIMITS_FILE=config/limits.alps.yaml; the default +# config/limits.yaml stays the local compose demo. +# +# Values come from the live validation run of 2026-11-03 +# (docs/hot-cache-v2.md §8): `debug` was UP with 24 nodes, `normal`/`low` +# with 922 each. Only `debug` is allowed: pilots are short and must not +# compete with production work. Keep these tighter than the template bounds. +max_nodes: 2 +max_time_minutes: 30 +allowed_partitions: + - debug +allowed_systems: + - daint +# Home directories live under /users/, personal scratch under +# /capstor/scratch/cscs/. Narrow to your own user if you can. +allowed_working_directories: + - /users + - /capstor/scratch/cscs diff --git a/docs/hot-cache-v2.md b/docs/hot-cache-v2.md index e44b07f..cc88ad3 100644 --- a/docs/hot-cache-v2.md +++ b/docs/hot-cache-v2.md @@ -19,6 +19,8 @@ picture of v2 before deciding how `firecrest-mcp/` should speak it. digest `sha256:3dedfc6415191b86386e688938f31c1f1204fb473e0c3318eb92aea2fbd0226d`, pulled 2026-09-12. The FirecREST app itself reported `app_version: 2.6.0` (`GET /openapi.json` → `info.version`). +- **Production Alps (2026-11-03):** first live run against + `api.svc.cscs.ch/hpc/firecrest/v2` (FirecREST 2.6.1, system `daint`) — see §8. - **Current stack:** FirecREST **2.6.1** (`ghcr.io/eth-cscs/firecrest-v2-demo:2.6.1`, index digest `sha256:ed12f77ea5f21964805562743f02e22e810fe742246826a01111f411b27f494c`, source tag commit `e6b5a418c7aa1f9f5bacea413883fc8a1f8eeb7c`). See `docs/local-env.md`. @@ -485,3 +487,97 @@ what the request implied. | `download_file(path, job_id)` | saving a job output | records path + SHA-256 in the job's card | | `get_provenance(job_id)` | "how was this run?" / reproduce | card in `runs/jobs//` (gitignored) | | `rerun_from_provenance(job_id)` | rerun a past job | checks rendered-script hash, then needs a new approval | + +## 8. Production Alps — first live validation, 2026-11-03 + +Everything above §8 was verified against the demo launcher or the local compose +stack. This section is the first run against **production Alps**, done during the +CSCS FirecREST Hackathon (OAT ETH Zürich) with the hackathon course credential. + +- **Endpoint:** `https://api.svc.cscs.ch/hpc/firecrest/v2`, FirecREST **2.6.1**. +- **Token:** CSCS Keycloak, realm `firecrest-clients`, client-credentials grant → + `200`, Bearer token with a **300 s** TTL. pyfirecrest's `ClientCredentialsAuth` + renews it before expiry, so `client_v2.py` needs nothing for long polls. +- **Systems visible:** `eiger`, `daint`. All runs below target `daint`. +- **How it was run:** direct HTTP calls (curl), one by one — **not** through the + MCP server. The prepare → approve → submit path of `server.py` has not yet run + on Alps; see "Not yet verified" below. + +### 8.1 Sequence and observed answers + +| # | Call | Observed | +|---|---|---| +| 1 | `GET /status/systems` without a token | `401`, proper challenge: gateway live | +| 2 | `POST` to Keycloak (client credentials) | `200`, Bearer, `expires_in` 300 | +| 3 | `GET /status/systems` | `eiger`, `daint`; scheduler, SSH, filesystem, S3 healthy | +| 4 | `GET /status/daint/userinfo` | user `course_941716d1`, group `f7t-hack-course2026-cscs` | +| 5 | `GET /status/daint/partitions` | `debug` 24 nodes, `low`/`normal` 922 each, `xfer` 2 — all `UP` | +| 6 | `GET /filesystem/daint/ops/ls`, `ops/stat` | home and scratch resolved; uid/gid match userinfo | +| 7 | `POST /compute/daint/jobs` (inline script, with `account`) | `201`, `{"jobId": "5245627"}` | +| 8 | `GET /compute/daint/jobs/5245627` | `status.state` `COMPLETED`, exit code `0`, node `nid006554`, elapsed 14 s | +| 9 | `GET /filesystem/daint/ops/view` on the stdout path | `hello from nid006554 at 2026-11-03T13:07:53+01:00` / `done` | + +### 8.2 Findings, and what they mean for `client_v2.py` + +1. **Payload shape is nested camelCase** (`{"job": {"script", "workingDirectory", ...}}`). + Same as §4. Only hand-written calls get this wrong; `client_v2.py` goes through + `pyfirecrest.v2.Firecrest.submit`, which builds exactly this body. + **No code change.** +2. **A Slurm project account is mandatory on Alps.** Without `job.account` the + answer is `500` with `"Unexpected Slurm command response ... you must specify a + project account"` — a scheduler refusal surfaced as a server error, not a `4xx`. + The valid value was `f7t-hack-course2026-cscs`; `root` (Slurm's cluster-level + association) is rejected as a non-existent project. **Changed:** + - `FIRECREST_V2_ACCOUNT` sets the account; `submit_job` falls back to it. + - Without it, `prepare_job` reads `userinfo.accounts` (FirecREST ≥ 2.6, + `sacctmgr`-backed, `UserInfoResponse` in `src/firecrest/status/models.py`) + and takes the only non-`root` account, or the one flagged default; when that + is ambiguous it leaves the account unset rather than guess whom to bill. + The account is resolved **before approval** and shown in the summary, so the + human approves what will be billed. + - That specific `500` now carries a hint naming `FIRECREST_V2_ACCOUNT`; the + response body is matched, never echoed. +3. **`GET /compute/{system}/jobs` lists only pending/running jobs.** A finished job + is read with `GET /compute/{system}/jobs/{id}`, state under `status.state`. + `client_v2.get_job_status` already uses `job_info(system, id)`, i.e. the per-id + route. **No code change.** Note Slurm's default 24 h accounting window: pass + `time_window` for older jobs. +4. **`GET /compute/{system}/jobs/{id}/metadata` answered `404`** for the completed + job. Upstream explains it (`src/lib/scheduler_clients/slurm/slurm_cli_client.py`, + `get_job_metadata`, eth-cscs/firecrest-v2 at `7616479`): paths come from + `scontrol show job`, which forgets a job minutes after it ends (Slurm + `MinJobAge`), and the `sacct` fallback needs Slurm ≥ 24.05 plus + `AccountingStoreFlags=job_env,job_script`. **Changed:** `get_job_log` already fell + back to `/slurm-.out`; it now also tries `slurm-.err` + (what the templates write) and skips it quietly if absent. `submit_job` keeps + deriving paths from the `#SBATCH --output/--error` directives when metadata is + unavailable. + +Items 2 and 4 are worth raising upstream: a missing account could be a `400` with +the Slurm message, and the metadata gap could be documented next to the endpoint. + +### 8.3 Running the workbench against Alps + +```bash +# firecrest-mcp/.env (never committed) +FIRECREST_API_VERSION=v2 +FIRECREST_V2_BASE_URL=https://api.svc.cscs.ch/hpc/firecrest/v2 +FIRECREST_V2_TOKEN_URL= +FIRECREST_V2_SYSTEM=daint +FIRECREST_V2_ACCOUNT= # optional, see 8.2 item 2 +FIRECREST_V2_WORKING_DIRECTORY=/users/ + +# guardrails for Alps instead of the compose demo +FCAGENT_LIMITS_FILE=config/limits.alps.yaml # daint, debug only, ≤2 nodes, ≤30 min +FCAGENT_TEMPLATE_DIR=templates/alps +``` + +### 8.4 Not yet verified on Alps + +- The MCP path end to end (`prepare_job` → `fcagent approve` → `submit_job` → + `get_job_status` → `get_job_log`) with the profile above. +- `ops/head`, which `get_job_log` uses (the manual run used `ops/view`). +- `cancel_job`, multi-node jobs, and `eiger`. + +Regression tests for this section: `firecrest-mcp/tests/test_client_v2.py` +(block "Production Alps") and `tests/test_alps_config.py`. diff --git a/firecrest-mcp/.env.example b/firecrest-mcp/.env.example index 63effc1..9323234 100644 --- a/firecrest-mcp/.env.example +++ b/firecrest-mcp/.env.example @@ -72,6 +72,19 @@ FIRECREST_V2_CLIENT_SECRET=replace-me # former. FIRECREST_V2_SYSTEM=cluster-slurm-api +# Slurm project account. Mandatory on production Alps (a job without one is +# refused with HTTP 500); not needed by the compose demo. Unset, prepare_job +# takes the single non-root project from /status/{system}/userinfo, or none. +# FIRECREST_V2_ACCOUNT= + +# Production Alps (validated 2026-11-03, docs/hot-cache-v2.md §8): +# FIRECREST_V2_BASE_URL=https://api.svc.cscs.ch/hpc/firecrest/v2 +# FIRECREST_V2_TOKEN_URL= +# FIRECREST_V2_SYSTEM=daint +# FIRECREST_V2_WORKING_DIRECTORY=/users/ +# and, for the guardrails, FCAGENT_LIMITS_FILE=config/limits.alps.yaml and +# FCAGENT_TEMPLATE_DIR=templates/alps. + # --- Optional --------------------------------------------------------------- FIRECREST_V2_TIMEOUT=30 diff --git a/firecrest-mcp/client_v2.py b/firecrest-mcp/client_v2.py index c41c2d6..37c7b14 100644 --- a/firecrest-mcp/client_v2.py +++ b/firecrest-mcp/client_v2.py @@ -82,6 +82,10 @@ class FirecRESTConfigV2: system: str #: Optional default for prepare_job; None means the caller must pass one. working_directory: str | None = None + #: Slurm project account. Mandatory on production Alps (a submission + #: without one is answered with HTTP 500, docs/hot-cache-v2.md §8); the + #: local compose demo does not need it. None means "resolve from userinfo". + account: str | None = None timeout_s: float = DEFAULT_TIMEOUT_S log_dir: Path = field(default_factory=lambda: Path("logs")) verify_ssl: bool = True @@ -104,6 +108,7 @@ def need(name: str) -> str: client_secret=need("FIRECREST_V2_CLIENT_SECRET"), system=need("FIRECREST_V2_SYSTEM"), working_directory=os.environ.get("FIRECREST_V2_WORKING_DIRECTORY") or None, + account=os.environ.get("FIRECREST_V2_ACCOUNT") or None, timeout_s=float(os.environ.get("FIRECREST_V2_TIMEOUT", DEFAULT_TIMEOUT_S)), log_dir=Path(os.environ.get("FIRECREST_V2_LOG_DIR", "logs")), verify_ssl=os.environ.get("FIRECREST_V2_VERIFY_SSL", "true").lower() @@ -176,6 +181,55 @@ def _http_status(exc: Exception) -> int | None: return getattr(responses[-1], "status_code", None) if responses else None +def _error_message(exc: Exception) -> str: + """The ``message`` of FirecREST's error envelope, for matching only. + + Never returned to the caller: it may echo paths or user names. + """ + responses = getattr(exc, "responses", None) + try: + body = responses[-1].json() if responses else None + except Exception: # noqa: BLE001 - non-JSON body + return "" + message = body.get("message") if isinstance(body, dict) else None + return message if isinstance(message, str) else "" + + +#: Observed on production Alps (FirecREST 2.6.1, 2026-11-03): a submission +#: without ``job.account`` is answered with HTTP 500 and the message +#: "Unexpected Slurm command response ... you must specify a project account". +_ACCOUNT_HINT = ( + "Slurm requires a project account on this system: set FIRECREST_V2_ACCOUNT " + "or pass account (see /status/{system}/userinfo)" +) + + +def _pick_account(userinfo: Any) -> str | None: + """Choose the Slurm account from a v2 ``userinfo`` answer, or ``None``. + + ``userinfo.accounts`` (FirecREST >= 2.6, from ``sacctmgr``) lists the + user's associations with a ``default`` flag. ``root`` is Slurm's + cluster-level association, never a project, and Alps rejects it. One + remaining candidate is used; with several, only the one flagged default; + otherwise ``None``, because guessing whom to bill is not acceptable. + """ + accounts = userinfo.get("accounts") if isinstance(userinfo, dict) else None + if not isinstance(accounts, list): + return None + candidates = [ + entry + for entry in accounts + if isinstance(entry, dict) + and isinstance(entry.get("name"), str) + and entry["name"] + and entry["name"] != "root" + ] + if len(candidates) == 1: + return candidates[0]["name"] + defaults = [entry for entry in candidates if entry.get("default") is True] + return defaults[0]["name"] if len(defaults) == 1 else None + + #: Cause hints for FirecREST v2 error statuses, from the upstream mapping in #: eth-cscs/firecrest-v2 tag 2.6.1, src/lib/models/apis/api_response_model.py:43-54 #: (TimeoutLimitExceeded -> 504; SSHClientError/SSHServiceError -> 502; @@ -265,6 +319,12 @@ def run() -> Any: except Exception as exc: status = _http_status(exc) hint = _STATUS_HINTS.get(status or 0) + if ( + method == "submit" + and status == 500 + and "project account" in _error_message(exc) + ): + hint = _ACCOUNT_HINT message = f"FirecREST v2 {method} failed" if hint: message = f"{message}: {hint}" @@ -274,6 +334,25 @@ async def list_systems(self) -> list[dict[str, Any]]: """Return systems using ``Firecrest.systems()``.""" return await self._call("systems") + async def resolve_account(self, system: str | None = None) -> str | None: + """The Slurm account to submit with, or ``None`` (never raises). + + ``FIRECREST_V2_ACCOUNT`` wins. Otherwise ``GET /status/{system}/userinfo`` + is read and :func:`_pick_account` decides. Called at prepare time so the + human approves the account that will actually be billed. + """ + configured = getattr(self.config, "account", None) + if configured: + return configured + trace = self.last_trace # a lookup must not replace the trace of a submit + try: + info = await self._call("userinfo", system or self.config.system) + except Exception: # noqa: BLE001 - e.g. demo scheduler without sacctmgr + return None + finally: + self.last_trace = trace + return _pick_account(info) + async def submit_job( self, script: str | None = None, @@ -292,6 +371,7 @@ async def submit_job( if not working_directory or not working_directory.startswith("/"): raise FirecRESTError("submit_job needs an absolute working_directory") system = system or self.config.system + account = account or getattr(self.config, "account", None) response = await self._call( "submit", system, @@ -428,8 +508,16 @@ async def get_job_log(self, job_id: str, system: str | None = None) -> JobLog: (``job_metadata`` answers 501 there — see docs/local-env.md), so when metadata yields no paths this falls back to Slurm's own default ``/slurm-.out``, same as scripts/smoke_local.py. + + Production Alps answers ``404`` for the metadata of a job Slurm has + already purged from ``scontrol`` (minutes after it ends) when ``sacct`` + does not store job scripts — see docs/hot-cache-v2.md §8. The same + fallback then applies, and also tries ``slurm-.err`` (what the + shipped templates write); a derived stderr path that cannot be read is + skipped, not an error. """ system = system or self.config.system + derived = False try: metadata = await self.get_job_metadata(job_id, system) except FirecRESTError: @@ -437,18 +525,26 @@ async def get_job_log(self, job_id: str, system: str | None = None) -> JobLog: if not metadata.get("standardOutput") and not metadata.get("standardError"): status = await self.get_job_status(job_id, system) if status.working_directory: + base = f"{status.working_directory.rstrip('/')}/slurm-{job_id}" metadata = { - "standardOutput": f"{status.working_directory}/slurm-{job_id}.out" + "standardOutput": f"{base}.out", + "standardError": f"{base}.err", } + derived = True log = JobLog(jobid=str(job_id)) for metadata_key, stream in ( ("standardOutput", "stdout"), ("standardError", "stderr"), ): if remote_path := metadata.get(metadata_key): - output = await self._call( - "head", system, remote_path, num_bytes=DEFAULT_OUTPUT_BYTES - ) + try: + output = await self._call( + "head", system, remote_path, num_bytes=DEFAULT_OUTPUT_BYTES + ) + except FirecRESTError: + if derived and stream == "stderr": + continue + raise text = ( output.get("content", "") if isinstance(output, dict) diff --git a/firecrest-mcp/server.py b/firecrest-mcp/server.py index bdcf405..afc1695 100644 --- a/firecrest-mcp/server.py +++ b/firecrest-mcp/server.py @@ -247,6 +247,9 @@ async def prepare_job( list_templates (`working_directory.default`); if that is `null` you must pass one. It must be a normalized absolute path under an allowed directory (also listed there); anything else is refused before anything can be approved. + `account` (Slurm project) is optional: omitted, it comes from + FIRECREST_V2_ACCOUNT or the user's single project in userinfo, and the + `summary` shows it. Production Alps refuses jobs without one. """ try: _, service = _core() @@ -258,6 +261,16 @@ async def prepare_job( ) if working_directory is None: raise GuardrailDenied("working_directory_invalid") + if account is None: + # Resolved before approval, so the human approves the account that + # will be billed (mandatory on Alps; None is fine on the demo). + # A client that cannot be built yet must not mask a guardrail denial. + try: + client = await get_client() + except Exception: # noqa: BLE001 + client = None + if isinstance(client, FirecRESTClientV2): + account = await client.resolve_account(system) prepared = service.prepare( actor=actor, system=system, diff --git a/firecrest-mcp/tests/test_client_v2.py b/firecrest-mcp/tests/test_client_v2.py index f89c161..2b96ed2 100644 --- a/firecrest-mcp/tests/test_client_v2.py +++ b/firecrest-mcp/tests/test_client_v2.py @@ -333,3 +333,233 @@ def test_shipped_templates_declare_output_paths_with_only_percent_j( assert derived["out"] == "/home/demo/slurm-42.out" assert derived["err"] == "/home/demo/slurm-42.err" assert derived["unsupported"] is False + + +# -------------------------------------------------------------------------- +# Production Alps (FirecREST 2.6.1, system daint), validated 2026-11-03. +# Shapes below mirror what that run returned; see docs/hot-cache-v2.md §8. +# -------------------------------------------------------------------------- + +ALPS_ACCOUNT = "f7t-hack-course2026-cscs" + + +def _alps_config(tmp_path, **over: Any) -> f7t2.FirecRESTConfigV2: + values: dict[str, Any] = { + "base_url": "https://api.svc.cscs.ch/hpc/firecrest/v2", + "token_url": "https://auth.test/token", + "client_id": "test-client", + "client_secret": "fictional-secret-must-never-appear", + "system": "daint", + "log_dir": tmp_path / "logs", + } + values.update(over) + return f7t2.FirecRESTConfigV2(**values) + + +class _Userinfo(FakeFirecrest): + def __init__(self, accounts: Any) -> None: + super().__init__() + self.accounts = accounts + + def userinfo(self, *args: Any) -> dict[str, Any]: + self._call("userinfo", *args) + info: dict[str, Any] = { + "user": {"id": "1000", "name": "course_941716d1"}, + "groups": [{"id": "1000", "name": ALPS_ACCOUNT, "default": True}], + } + if self.accounts is not None: + info["accounts"] = self.accounts + return info + + +def test_account_is_read_from_env(monkeypatch, tmp_path) -> None: + for name, value in { + "FIRECREST_V2_BASE_URL": "https://firecrest.test", + "FIRECREST_V2_TOKEN_URL": "https://auth.test/token", + "FIRECREST_V2_CLIENT_ID": "id", + "FIRECREST_V2_CLIENT_SECRET": "s", + "FIRECREST_V2_SYSTEM": "daint", + "FIRECREST_V2_ACCOUNT": ALPS_ACCOUNT, + }.items(): + monkeypatch.setenv(name, value) + cfg = f7t2.FirecRESTConfigV2.from_env(tmp_path / "missing.env") + assert cfg.account == ALPS_ACCOUNT + monkeypatch.delenv("FIRECREST_V2_ACCOUNT") + assert f7t2.FirecRESTConfigV2.from_env(tmp_path / "missing.env").account is None + + +async def test_submit_uses_configured_account_and_explicit_one_wins(tmp_path) -> None: + fake = FakeFirecrest() + client = f7t2.FirecRESTClientV2( + _alps_config(tmp_path, account=ALPS_ACCOUNT), factory=lambda _: fake + ) + await client.submit_job("#!/bin/bash\n", working_directory="/users/u") + await client.submit_job("#!/bin/bash\n", working_directory="/users/u", account="p1") + accounts = [c[2]["account"] for c in fake.calls if c[0] == "submit"] + assert accounts == [ALPS_ACCOUNT, "p1"] + + +async def test_resolve_account_prefers_config_without_a_remote_call(tmp_path) -> None: + fake = _Userinfo([]) + client = f7t2.FirecRESTClientV2( + _alps_config(tmp_path, account="from-env"), factory=lambda _: fake + ) + assert await client.resolve_account() == "from-env" + assert not fake.calls + + +@pytest.mark.parametrize( + ("accounts", "expected"), + [ + # root is Slurm's cluster association; Alps rejects it as a project. + ( + [ + {"name": "root", "default": True}, + {"name": ALPS_ACCOUNT, "default": False}, + ], + ALPS_ACCOUNT, + ), + ([{"name": ALPS_ACCOUNT, "default": True}], ALPS_ACCOUNT), + ([{"name": "p1", "default": False}, {"name": "p2", "default": True}], "p2"), + # Ambiguous or empty: never guess whom to bill. + ([{"name": "p1", "default": False}, {"name": "p2", "default": False}], None), + ([{"name": "root", "default": True}], None), + ([], None), + (None, None), # FirecREST < 2.6 has no `accounts` field + ], +) +async def test_resolve_account_from_userinfo(tmp_path, accounts, expected) -> None: + fake = _Userinfo(accounts) + client = f7t2.FirecRESTClientV2(_alps_config(tmp_path), factory=lambda _: fake) + client.last_trace = {"x_correlation_id": "submit-cid"} + assert await client.resolve_account() == expected + assert fake.calls == [("userinfo", ("daint",), {})] + assert client.last_trace == {"x_correlation_id": "submit-cid"} + + +async def test_resolve_account_failure_is_none_not_an_error(tmp_path) -> None: + class NoSacctmgr(FakeFirecrest): + def userinfo(self, *_: Any) -> dict[str, Any]: + raise _HttpError(500) + + client = f7t2.FirecRESTClientV2( + _alps_config(tmp_path), factory=lambda _: NoSacctmgr() + ) + assert await client.resolve_account() is None + + +class _JsonResp: + def __init__(self, status_code: int, body: Any) -> None: + self.status_code = status_code + self._body = body + + def json(self) -> Any: + return self._body + + +class _EnvelopeError(Exception): + def __init__(self, status_code: int, message: str) -> None: + self.responses = [ + _JsonResp(status_code, {"errorType": "error", "message": message}) + ] + + +async def test_missing_account_500_on_alps_gets_an_actionable_hint(tmp_path) -> None: + message = ( + "Unexpected Slurm command response. exit_status:1 std_err:sbatch: error: " + "you must specify a project account (course_941716d1)" + ) + + class Alps(FakeFirecrest): + def submit(self, *_: Any, **__: Any) -> dict[str, str]: + raise _EnvelopeError(500, message) + + client = f7t2.FirecRESTClientV2(_alps_config(tmp_path), factory=lambda _: Alps()) + with pytest.raises(f7t2.FirecRESTError) as info: + await client.submit_job("#!/bin/bash\n", working_directory="/users/u") + assert info.value.status == 500 + assert "FIRECREST_V2_ACCOUNT" in str(info.value) + assert "course_941716d1" not in str(info.value) # body is matched, never echoed + + +async def test_other_500_on_submit_gets_no_account_hint(tmp_path) -> None: + class Broken(FakeFirecrest): + def submit(self, *_: Any, **__: Any) -> dict[str, str]: + raise _EnvelopeError(500, "Unexpected Slurm command response") + + client = f7t2.FirecRESTClientV2(_alps_config(tmp_path), factory=lambda _: Broken()) + with pytest.raises(f7t2.FirecRESTError) as info: + await client.submit_job("#!/bin/bash\n", working_directory="/users/u") + assert "FIRECREST_V2_ACCOUNT" not in str(info.value) + + +class _AlpsCompleted(FakeFirecrest): + """A finished job on Alps: /metadata is 404, /jobs/{id} still answers.""" + + def __init__(self, files: dict[str, str]) -> None: + super().__init__() + self.files = files + + def job_metadata(self, *args: Any) -> list[dict[str, str]]: + self._call("job_metadata", *args) + raise _HttpError(404) + + def job_info(self, *args: Any, **kwargs: Any) -> list[dict[str, Any]]: + self._call("job_info", *args, **kwargs) + return [ + { + "jobId": "5245627", + "status": {"state": "COMPLETED", "exitCode": 0}, + "nodes": "nid006554", + "partition": "debug", + "workingDirectory": "/users/course_941716d1/", + "time": {"elapsed": 14}, + } + ] + + def head(self, system: str, path: str, **kwargs: Any) -> dict[str, str]: + self._call("head", system, path, **kwargs) + if path not in self.files: + raise _HttpError(404) + return {"content": self.files[path]} + + +async def test_completed_alps_job_state_comes_from_job_info_by_id(tmp_path) -> None: + fake = _AlpsCompleted({}) + client = f7t2.FirecRESTClientV2(_alps_config(tmp_path), factory=lambda _: fake) + status = await client.get_job_status("5245627") + assert (status.state, status.exit_code, status.nodes) == ( + "COMPLETED", + 0, + "nid006554", + ) + assert fake.calls == [("job_info", ("daint", "5245627"), {})] + + +async def test_alps_metadata_404_falls_back_to_slurm_out_and_err(tmp_path) -> None: + base = "/users/course_941716d1/slurm-5245627" + fake = _AlpsCompleted( + {f"{base}.out": "hello from nid006554\ndone\n", f"{base}.err": ""} + ) + client = f7t2.FirecRESTClientV2(_alps_config(tmp_path), factory=lambda _: fake) + log = await client.get_job_log("5245627") + assert log.stdout == "hello from nid006554\ndone\n" + assert log.stderr == "" + heads = [c[1][1] for c in fake.calls if c[0] == "head"] + assert heads == [f"{base}.out", f"{base}.err"] # no double slash + + +async def test_alps_fallback_skips_a_missing_derived_stderr(tmp_path) -> None: + base = "/users/course_941716d1/slurm-5245627" + fake = _AlpsCompleted({f"{base}.out": "done\n"}) + client = f7t2.FirecRESTClientV2(_alps_config(tmp_path), factory=lambda _: fake) + log = await client.get_job_log("5245627") + assert log.stdout == "done\n" and log.stderr is None + + +async def test_alps_fallback_still_fails_when_stdout_is_missing(tmp_path) -> None: + client = f7t2.FirecRESTClientV2( + _alps_config(tmp_path), factory=lambda _: _AlpsCompleted({}) + ) + with pytest.raises(f7t2.FirecRESTError): + await client.get_job_log("5245627") diff --git a/firecrest-mcp/tests/test_server_submit.py b/firecrest-mcp/tests/test_server_submit.py index f887c82..2c850e0 100644 --- a/firecrest-mcp/tests/test_server_submit.py +++ b/firecrest-mcp/tests/test_server_submit.py @@ -636,3 +636,46 @@ async def test_summary_never_invents_missing_values() -> None: assert s["workbench_commit"] is None and s["outputs"] == [] assert s["approved_by"] is None and "rerun_of" not in s assert summarize({"job_id": "2", "rerun_of": "1"})["rerun_of"] == "1" + + +class AlpsLikeV2(FakeV2): + """Account resolved from userinfo, as on production Alps (2026-11-03).""" + + def __init__(self) -> None: + super().__init__() + self.resolved: list[str] = [] + + async def resolve_account(self, system: str | None = None) -> str | None: # type: ignore[override] + self.resolved.append(system or "") + return "f7t-hack-course2026-cscs" + + +async def test_prepare_resolves_account_so_the_human_approves_it( + env: SimpleNamespace, monkeypatch: pytest.MonkeyPatch +) -> None: + fake = AlpsLikeV2() + + async def client(): + return fake + + monkeypatch.setattr(server, "get_client", client) + prepared = await _prepare() + assert "account f7t-hack-course2026-cscs" in prepared["summary"] + assert fake.resolved == ["cluster-slurm-api"] + _approve(env, prepared["prepared_id"]) + result = await server.submit_job(prepared_id=prepared["prepared_id"]) + assert result["ok"] is True + assert fake.calls[0]["account"] == "f7t-hack-course2026-cscs" + + +async def test_explicit_account_skips_resolution( + env: SimpleNamespace, monkeypatch: pytest.MonkeyPatch +) -> None: + fake = AlpsLikeV2() + + async def client(): + return fake + + monkeypatch.setattr(server, "get_client", client) + prepared = await _prepare(account="proj-x") + assert "account proj-x" in prepared["summary"] and fake.resolved == [] diff --git a/src/fcagent/core/jobs.py b/src/fcagent/core/jobs.py index 83d41df..1a5ea0c 100644 --- a/src/fcagent/core/jobs.py +++ b/src/fcagent/core/jobs.py @@ -402,6 +402,7 @@ def summarize(context: dict[str, Any]) -> str: f"time={context['parameters'].get('time_minutes')} min, " f"estimated {context['node_hours']} node-hours; " f"working directory {context['working_directory']}; " - f"valid until {context['expires_at']}; " + + (f"account {context['account']}; " if context.get("account") else "") + + f"valid until {context['expires_at']}; " f"script sha256 {context['script_sha256'][:16]}…" ) diff --git a/templates/alps/hello.sh b/templates/alps/hello.sh new file mode 100644 index 0000000..89036aa --- /dev/null +++ b/templates/alps/hello.sh @@ -0,0 +1,8 @@ +#!/bin/bash +#SBATCH --nodes={{nodes}} +#SBATCH --time={{time_minutes}} +#SBATCH --partition={{partition}} +#SBATCH --job-name={{job_name}} +#SBATCH --output=slurm-%j.out +#SBATCH --error=slurm-%j.err +echo "hello-from-fcagent {{message}}" diff --git a/templates/alps/hello.yaml b/templates/alps/hello.yaml new file mode 100644 index 0000000..6aec304 --- /dev/null +++ b/templates/alps/hello.yaml @@ -0,0 +1,19 @@ +# Alps (daint) variant of templates/hello.yaml, used with +# FCAGENT_TEMPLATE_DIR=templates/alps and FCAGENT_LIMITS_FILE=config/limits.alps.yaml. +# Same script; only the partition and the bounds differ (debug partition only). +parameters: + nodes: + type: int + min: 1 + max: 2 + time_minutes: + type: int + min: 1 + max: 30 + partition: + type: enum + values: [debug] + job_name: + type: string + message: + type: string diff --git a/tests/test_alps_config.py b/tests/test_alps_config.py new file mode 100644 index 0000000..c1f535d --- /dev/null +++ b/tests/test_alps_config.py @@ -0,0 +1,84 @@ +"""The shipped Alps profile (config/limits.alps.yaml + templates/alps/). + +Values were validated live on production Alps, system daint, on 2026-11-03 +(docs/hot-cache-v2.md §8). No network: only the guardrails are exercised. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +import yaml + +from fcagent.core.audit import AuditLog +from fcagent.core.jobs import GuardrailDenied, JobService +from fcagent.core.settings import Limits + +REPO = Path(__file__).resolve().parent.parent +ALPS_LIMITS = REPO / "config/limits.alps.yaml" +ALPS_TEMPLATES = REPO / "templates/alps" +PARAMS = { + "nodes": 1, + "time_minutes": 5, + "partition": "debug", + "job_name": "alps-hello", + "message": "hello", +} + + +def _service(tmp_path: Path) -> JobService: + return JobService( + template_directory=ALPS_TEMPLATES, + limits=Limits.from_yaml(ALPS_LIMITS), + audit=AuditLog(tmp_path / "audit.jsonl"), + client=None, + ) + + +def test_alps_templates_stay_within_alps_limits() -> None: + limits = yaml.safe_load(ALPS_LIMITS.read_text()) + for schema in ALPS_TEMPLATES.glob("*.yaml"): + spec = yaml.safe_load(schema.read_text())["parameters"] + assert spec["nodes"]["max"] <= limits["max_nodes"] + assert spec["time_minutes"]["max"] <= limits["max_time_minutes"] + assert set(spec["partition"]["values"]) <= set(limits["allowed_partitions"]) + script = schema.with_suffix(".sh").read_text() + assert "--output=slurm-%j.out" in script # what get_job_log falls back to + for name in spec: + assert "{{" + name + "}}" in script + + +def test_alps_hello_renders_for_daint_debug(tmp_path: Path) -> None: + service = _service(tmp_path) + service._check_system("daint") + service._check_working_directory("/users/course_941716d1") + _, script = service.render( + template="hello", + parameters=dict(PARAMS), + working_directory="/users/course_941716d1", + ) + assert "#SBATCH --partition=debug" in script + + +@pytest.mark.parametrize( + "override", + [{"partition": "normal"}, {"nodes": 3}, {"time_minutes": 31}], +) +def test_alps_profile_refuses_production_partitions_and_big_jobs( + tmp_path: Path, override: dict +) -> None: + with pytest.raises(GuardrailDenied): + _service(tmp_path).render( + template="hello", + parameters={**PARAMS, **override}, + working_directory="/users/course_941716d1", + ) + + +def test_alps_profile_refuses_other_systems_and_paths(tmp_path: Path) -> None: + service = _service(tmp_path) + with pytest.raises(GuardrailDenied): + service._check_system("cluster-slurm-api") + with pytest.raises(GuardrailDenied): + service._check_working_directory("/tmp")