Alps validation 2026-11-03: Slurm account, metadata 404 fallback, Alps profile

CSCS-PROPOSAL.md

@@ -2,6 +2,16 @@
 
 > **Status: participation confirmed.** Event on 3 November 2026, OAT ETH Zürich (rooms S15/S16, 14th floor). This document and the rest of the repo are what we're bringing to build on the day — see `prompts/` for the build order and `README.md` → Status for logistics.
 
+## Update — 3 November 2026: first run on production Alps
+
+During the hackathon the full FirecREST v2 path — token, system discovery,
+filesystem, job submission, polling and output — was exercised against
+production Alps (`api.svc.cscs.ch/hpc/firecrest/v2`, FirecREST 2.6.1, system
+`daint`): job `5245627` ran on `nid006554` and exited `0`. Findings and the
+resulting client changes (mandatory Slurm account, `/metadata` 404 fallback) are
+in [`docs/hot-cache-v2.md` §8](./docs/hot-cache-v2.md); the Alps guardrail
+profile is `config/limits.alps.yaml` + `templates/alps/`.
+
 ## One sentence
 
 Let a researcher submit, monitor and diagnose HPC jobs through natural language — from a chat app, not a terminal — with the whole stack running on CSCS's own open-model inference service, not a third-party API.

README.md

@@ -196,6 +196,7 @@ human approval gates for `sudo` and public exposure, and verifies every phase.
 | 06 | Multica orchestration | done and verified live | [`MULTICA.md`](./docs/MULTICA.md) |
 | 07 | Telegram channel | not started — requires a human to provide a BotFather token and bind the workspace channel | — |
 | 08 | Failure scenario | partial delivery — recorded directly against FirecREST; it still needs a Hermes re-run | [`failure-scenario-transcript.md`](./docs/failure-scenario-transcript.md) |
+| — | **Production Alps (FirecREST 2.6.1, `daint`)** | auth → submit → poll → output verified live on 2026-11-03 via direct API calls; MCP path on Alps not yet run | [`hot-cache-v2.md` §8](./docs/hot-cache-v2.md#8-production-alps--first-live-validation-2026-11-03) |
 
 The MetaMCP endpoint currently aggregates six tools: five from `firecrest-mcp`
 and `docmind__query_docs` from the local RAG corpus. Resume from

config/limits.alps.yaml

@@ -0,0 +1,19 @@
+# Limits for production Alps (CSCS), system daint, through FirecREST v2.
+# Select with FCAGENT_LIMITS_FILE=config/limits.alps.yaml; the default
+# config/limits.yaml stays the local compose demo.
+#
+# Values come from the live validation run of 2026-11-03
+# (docs/hot-cache-v2.md §8): `debug` was UP with 24 nodes, `normal`/`low`
+# with 922 each. Only `debug` is allowed: pilots are short and must not
+# compete with production work. Keep these tighter than the template bounds.
+max_nodes: 2
+max_time_minutes: 30
+allowed_partitions:
+  - debug
+allowed_systems:
+  - daint
+# Home directories live under /users/<user>, personal scratch under
+# /capstor/scratch/cscs/<user>. Narrow to your own user if you can.
+allowed_working_directories:
+  - /users
+  - /capstor/scratch/cscs

docs/hot-cache-v2.md

@@ -19,6 +19,8 @@ picture of v2 before deciding how `firecrest-mcp/` should speak it.
   digest `sha256:3dedfc6415191b86386e688938f31c1f1204fb473e0c3318eb92aea2fbd0226d`,
   pulled 2026-09-12. The FirecREST app itself reported `app_version: 2.6.0`
   (`GET /openapi.json` → `info.version`).
+- **Production Alps (2026-11-03):** first live run against
+  `api.svc.cscs.ch/hpc/firecrest/v2` (FirecREST 2.6.1, system `daint`) — see §8.
 - **Current stack:** FirecREST **2.6.1** (`ghcr.io/eth-cscs/firecrest-v2-demo:2.6.1`,
   index digest `sha256:ed12f77ea5f21964805562743f02e22e810fe742246826a01111f411b27f494c`,
   source tag commit `e6b5a418c7aa1f9f5bacea413883fc8a1f8eeb7c`). See `docs/local-env.md`.
@@ -485,3 +487,97 @@ what the request implied.
 | `download_file(path, job_id)` | saving a job output | records path + SHA-256 in the job's card |
 | `get_provenance(job_id)` | "how was this run?" / reproduce | card in `runs/jobs/<jobId>/` (gitignored) |
 | `rerun_from_provenance(job_id)` | rerun a past job | checks rendered-script hash, then needs a new approval |
+
+## 8. Production Alps — first live validation, 2026-11-03
+
+Everything above §8 was verified against the demo launcher or the local compose
+stack. This section is the first run against **production Alps**, done during the
+CSCS FirecREST Hackathon (OAT ETH Zürich) with the hackathon course credential.
+
+- **Endpoint:** `https://api.svc.cscs.ch/hpc/firecrest/v2`, FirecREST **2.6.1**.
+- **Token:** CSCS Keycloak, realm `firecrest-clients`, client-credentials grant →
+  `200`, Bearer token with a **300 s** TTL. pyfirecrest's `ClientCredentialsAuth`
+  renews it before expiry, so `client_v2.py` needs nothing for long polls.
+- **Systems visible:** `eiger`, `daint`. All runs below target `daint`.
+- **How it was run:** direct HTTP calls (curl), one by one — **not** through the
+  MCP server. The prepare → approve → submit path of `server.py` has not yet run
+  on Alps; see "Not yet verified" below.
+
+### 8.1 Sequence and observed answers
+
+| # | Call | Observed |
+|---|---|---|
+| 1 | `GET /status/systems` without a token | `401`, proper challenge: gateway live |
+| 2 | `POST` to Keycloak (client credentials) | `200`, Bearer, `expires_in` 300 |
+| 3 | `GET /status/systems` | `eiger`, `daint`; scheduler, SSH, filesystem, S3 healthy |
+| 4 | `GET /status/daint/userinfo` | user `course_941716d1`, group `f7t-hack-course2026-cscs` |
+| 5 | `GET /status/daint/partitions` | `debug` 24 nodes, `low`/`normal` 922 each, `xfer` 2 — all `UP` |
+| 6 | `GET /filesystem/daint/ops/ls`, `ops/stat` | home and scratch resolved; uid/gid match userinfo |
+| 7 | `POST /compute/daint/jobs` (inline script, with `account`) | `201`, `{"jobId": "5245627"}` |
+| 8 | `GET /compute/daint/jobs/5245627` | `status.state` `COMPLETED`, exit code `0`, node `nid006554`, elapsed 14 s |
+| 9 | `GET /filesystem/daint/ops/view` on the stdout path | `hello from nid006554 at 2026-11-03T13:07:53+01:00` / `done` |
+
+### 8.2 Findings, and what they mean for `client_v2.py`
+
+1. **Payload shape is nested camelCase** (`{"job": {"script", "workingDirectory", ...}}`).
+   Same as §4. Only hand-written calls get this wrong; `client_v2.py` goes through
+   `pyfirecrest.v2.Firecrest.submit`, which builds exactly this body.
+   **No code change.**
+2. **A Slurm project account is mandatory on Alps.** Without `job.account` the
+   answer is `500` with `"Unexpected Slurm command response ... you must specify a
+   project account"` — a scheduler refusal surfaced as a server error, not a `4xx`.
+   The valid value was `f7t-hack-course2026-cscs`; `root` (Slurm's cluster-level
+   association) is rejected as a non-existent project. **Changed:**
+   - `FIRECREST_V2_ACCOUNT` sets the account; `submit_job` falls back to it.
+   - Without it, `prepare_job` reads `userinfo.accounts` (FirecREST ≥ 2.6,
+     `sacctmgr`-backed, `UserInfoResponse` in `src/firecrest/status/models.py`)
+     and takes the only non-`root` account, or the one flagged default; when that
+     is ambiguous it leaves the account unset rather than guess whom to bill.
+     The account is resolved **before approval** and shown in the summary, so the
+     human approves what will be billed.
+   - That specific `500` now carries a hint naming `FIRECREST_V2_ACCOUNT`; the
+     response body is matched, never echoed.
+3. **`GET /compute/{system}/jobs` lists only pending/running jobs.** A finished job
+   is read with `GET /compute/{system}/jobs/{id}`, state under `status.state`.
+   `client_v2.get_job_status` already uses `job_info(system, id)`, i.e. the per-id
+   route. **No code change.** Note Slurm's default 24 h accounting window: pass
+   `time_window` for older jobs.
+4. **`GET /compute/{system}/jobs/{id}/metadata` answered `404`** for the completed
+   job. Upstream explains it (`src/lib/scheduler_clients/slurm/slurm_cli_client.py`,
+   `get_job_metadata`, eth-cscs/firecrest-v2 at `7616479`): paths come from
+   `scontrol show job`, which forgets a job minutes after it ends (Slurm
+   `MinJobAge`), and the `sacct` fallback needs Slurm ≥ 24.05 plus
+   `AccountingStoreFlags=job_env,job_script`. **Changed:** `get_job_log` already fell
+   back to `<workingDirectory>/slurm-<id>.out`; it now also tries `slurm-<id>.err`
+   (what the templates write) and skips it quietly if absent. `submit_job` keeps
+   deriving paths from the `#SBATCH --output/--error` directives when metadata is
+   unavailable.
+
+Items 2 and 4 are worth raising upstream: a missing account could be a `400` with
+the Slurm message, and the metadata gap could be documented next to the endpoint.
+
+### 8.3 Running the workbench against Alps
+
+```bash
+# firecrest-mcp/.env (never committed)
+FIRECREST_API_VERSION=v2
+FIRECREST_V2_BASE_URL=https://api.svc.cscs.ch/hpc/firecrest/v2
+FIRECREST_V2_TOKEN_URL=<CSCS Keycloak token URL, realm firecrest-clients>
+FIRECREST_V2_SYSTEM=daint
+FIRECREST_V2_ACCOUNT=<your Slurm project>        # optional, see 8.2 item 2
+FIRECREST_V2_WORKING_DIRECTORY=/users/<user>
+
+# guardrails for Alps instead of the compose demo
+FCAGENT_LIMITS_FILE=config/limits.alps.yaml     # daint, debug only, ≤2 nodes, ≤30 min
+FCAGENT_TEMPLATE_DIR=templates/alps
+```
+
+### 8.4 Not yet verified on Alps
+
+- The MCP path end to end (`prepare_job` → `fcagent approve` → `submit_job` →
+  `get_job_status` → `get_job_log`) with the profile above.
+- `ops/head`, which `get_job_log` uses (the manual run used `ops/view`).
+- `cancel_job`, multi-node jobs, and `eiger`.
+
+Regression tests for this section: `firecrest-mcp/tests/test_client_v2.py`
+(block "Production Alps") and `tests/test_alps_config.py`.

firecrest-mcp/.env.example

@@ -72,6 +72,19 @@ FIRECREST_V2_CLIENT_SECRET=replace-me
 # former.
 FIRECREST_V2_SYSTEM=cluster-slurm-api
 
+# Slurm project account. Mandatory on production Alps (a job without one is
+# refused with HTTP 500); not needed by the compose demo. Unset, prepare_job
+# takes the single non-root project from /status/{system}/userinfo, or none.
+# FIRECREST_V2_ACCOUNT=
+
+# Production Alps (validated 2026-11-03, docs/hot-cache-v2.md §8):
+#   FIRECREST_V2_BASE_URL=https://api.svc.cscs.ch/hpc/firecrest/v2
+#   FIRECREST_V2_TOKEN_URL=<CSCS Keycloak token URL, realm firecrest-clients>
+#   FIRECREST_V2_SYSTEM=daint
+#   FIRECREST_V2_WORKING_DIRECTORY=/users/<user>
+# and, for the guardrails, FCAGENT_LIMITS_FILE=config/limits.alps.yaml and
+# FCAGENT_TEMPLATE_DIR=templates/alps.
+
 # --- Optional ---------------------------------------------------------------
 
 FIRECREST_V2_TIMEOUT=30

firecrest-mcp/client_v2.py

@@ -82,6 +82,10 @@ class FirecRESTConfigV2:
     system: str
     #: Optional default for prepare_job; None means the caller must pass one.
     working_directory: str | None = None
+    #: Slurm project account. Mandatory on production Alps (a submission
+    #: without one is answered with HTTP 500, docs/hot-cache-v2.md §8); the
+    #: local compose demo does not need it. None means "resolve from userinfo".
+    account: str | None = None
     timeout_s: float = DEFAULT_TIMEOUT_S
     log_dir: Path = field(default_factory=lambda: Path("logs"))
     verify_ssl: bool = True
@@ -104,6 +108,7 @@ def need(name: str) -> str:
             client_secret=need("FIRECREST_V2_CLIENT_SECRET"),
             system=need("FIRECREST_V2_SYSTEM"),
             working_directory=os.environ.get("FIRECREST_V2_WORKING_DIRECTORY") or None,
+            account=os.environ.get("FIRECREST_V2_ACCOUNT") or None,
             timeout_s=float(os.environ.get("FIRECREST_V2_TIMEOUT", DEFAULT_TIMEOUT_S)),
             log_dir=Path(os.environ.get("FIRECREST_V2_LOG_DIR", "logs")),
             verify_ssl=os.environ.get("FIRECREST_V2_VERIFY_SSL", "true").lower()
@@ -176,6 +181,55 @@ def _http_status(exc: Exception) -> int | None:
     return getattr(responses[-1], "status_code", None) if responses else None
 
 
+def _error_message(exc: Exception) -> str:
+    """The ``message`` of FirecREST's error envelope, for matching only.
+
+    Never returned to the caller: it may echo paths or user names.
+    """
+    responses = getattr(exc, "responses", None)
+    try:
+        body = responses[-1].json() if responses else None
+    except Exception:  # noqa: BLE001 - non-JSON body
+        return ""
+    message = body.get("message") if isinstance(body, dict) else None
+    return message if isinstance(message, str) else ""
+
+
+#: Observed on production Alps (FirecREST 2.6.1, 2026-11-03): a submission
+#: without ``job.account`` is answered with HTTP 500 and the message
+#: "Unexpected Slurm command response ... you must specify a project account".
+_ACCOUNT_HINT = (
+    "Slurm requires a project account on this system: set FIRECREST_V2_ACCOUNT "
+    "or pass account (see /status/{system}/userinfo)"
+)
+
+
+def _pick_account(userinfo: Any) -> str | None:
+    """Choose the Slurm account from a v2 ``userinfo`` answer, or ``None``.
+
+    ``userinfo.accounts`` (FirecREST >= 2.6, from ``sacctmgr``) lists the
+    user's associations with a ``default`` flag. ``root`` is Slurm's
+    cluster-level association, never a project, and Alps rejects it. One
+    remaining candidate is used; with several, only the one flagged default;
+    otherwise ``None``, because guessing whom to bill is not acceptable.
+    """
+    accounts = userinfo.get("accounts") if isinstance(userinfo, dict) else None
+    if not isinstance(accounts, list):
+        return None
+    candidates = [
+        entry
+        for entry in accounts
+        if isinstance(entry, dict)
+        and isinstance(entry.get("name"), str)
+        and entry["name"]
+        and entry["name"] != "root"
+    ]
+    if len(candidates) == 1:
+        return candidates[0]["name"]
+    defaults = [entry for entry in candidates if entry.get("default") is True]
+    return defaults[0]["name"] if len(defaults) == 1 else None
+
+
 #: Cause hints for FirecREST v2 error statuses, from the upstream mapping in
 #: eth-cscs/firecrest-v2 tag 2.6.1, src/lib/models/apis/api_response_model.py:43-54
 #: (TimeoutLimitExceeded -> 504; SSHClientError/SSHServiceError -> 502;
@@ -265,6 +319,12 @@ def run() -> Any:
         except Exception as exc:
             status = _http_status(exc)
             hint = _STATUS_HINTS.get(status or 0)
+            if (
+                method == "submit"
+                and status == 500
+                and "project account" in _error_message(exc)
+            ):
+                hint = _ACCOUNT_HINT
             message = f"FirecREST v2 {method} failed"
             if hint:
                 message = f"{message}: {hint}"
@@ -274,6 +334,25 @@ async def list_systems(self) -> list[dict[str, Any]]:
         """Return systems using ``Firecrest.systems()``."""
         return await self._call("systems")
 
+    async def resolve_account(self, system: str | None = None) -> str | None:
+        """The Slurm account to submit with, or ``None`` (never raises).
+
+        ``FIRECREST_V2_ACCOUNT`` wins. Otherwise ``GET /status/{system}/userinfo``
+        is read and :func:`_pick_account` decides. Called at prepare time so the
+        human approves the account that will actually be billed.
+        """
+        configured = getattr(self.config, "account", None)
+        if configured:
+            return configured
+        trace = self.last_trace  # a lookup must not replace the trace of a submit
+        try:
+            info = await self._call("userinfo", system or self.config.system)
+        except Exception:  # noqa: BLE001 - e.g. demo scheduler without sacctmgr
+            return None
+        finally:
+            self.last_trace = trace
+        return _pick_account(info)
+
     async def submit_job(
         self,
         script: str | None = None,
@@ -292,6 +371,7 @@ async def submit_job(
         if not working_directory or not working_directory.startswith("/"):
             raise FirecRESTError("submit_job needs an absolute working_directory")
         system = system or self.config.system
+        account = account or getattr(self.config, "account", None)
         response = await self._call(
             "submit",
             system,
@@ -428,27 +508,43 @@ async def get_job_log(self, job_id: str, system: str | None = None) -> JobLog:
         (``job_metadata`` answers 501 there — see docs/local-env.md), so when
         metadata yields no paths this falls back to Slurm's own default
         ``<working_directory>/slurm-<jobid>.out``, same as scripts/smoke_local.py.
+
+        Production Alps answers ``404`` for the metadata of a job Slurm has
+        already purged from ``scontrol`` (minutes after it ends) when ``sacct``
+        does not store job scripts — see docs/hot-cache-v2.md §8. The same
+        fallback then applies, and also tries ``slurm-<jobid>.err`` (what the
+        shipped templates write); a derived stderr path that cannot be read is
+        skipped, not an error.
         """
         system = system or self.config.system
+        derived = False
         try:
             metadata = await self.get_job_metadata(job_id, system)
         except FirecRESTError:
             metadata = {}
         if not metadata.get("standardOutput") and not metadata.get("standardError"):
             status = await self.get_job_status(job_id, system)
             if status.working_directory:
+                base = f"{status.working_directory.rstrip('/')}/slurm-{job_id}"
                 metadata = {
-                    "standardOutput": f"{status.working_directory}/slurm-{job_id}.out"
+                    "standardOutput": f"{base}.out",
+                    "standardError": f"{base}.err",
                 }
+                derived = True
         log = JobLog(jobid=str(job_id))
         for metadata_key, stream in (
             ("standardOutput", "stdout"),
             ("standardError", "stderr"),
         ):
             if remote_path := metadata.get(metadata_key):
-                output = await self._call(
-                    "head", system, remote_path, num_bytes=DEFAULT_OUTPUT_BYTES
-                )
+                try:
+                    output = await self._call(
+                        "head", system, remote_path, num_bytes=DEFAULT_OUTPUT_BYTES
+                    )
+                except FirecRESTError:
+                    if derived and stream == "stderr":
+                        continue
+                    raise
                 text = (
                     output.get("content", "")
                     if isinstance(output, dict)

firecrest-mcp/server.py

@@ -247,6 +247,9 @@ async def prepare_job(
     list_templates (`working_directory.default`); if that is `null` you must pass
     one. It must be a normalized absolute path under an allowed directory (also
     listed there); anything else is refused before anything can be approved.
+    `account` (Slurm project) is optional: omitted, it comes from
+    FIRECREST_V2_ACCOUNT or the user's single project in userinfo, and the
+    `summary` shows it. Production Alps refuses jobs without one.
     """
     try:
         _, service = _core()
@@ -258,6 +261,16 @@ async def prepare_job(
             )
             if working_directory is None:
                 raise GuardrailDenied("working_directory_invalid")
+        if account is None:
+            # Resolved before approval, so the human approves the account that
+            # will be billed (mandatory on Alps; None is fine on the demo).
+            # A client that cannot be built yet must not mask a guardrail denial.
+            try:
+                client = await get_client()
+            except Exception:  # noqa: BLE001
+                client = None
+            if isinstance(client, FirecRESTClientV2):
+                account = await client.resolve_account(system)
         prepared = service.prepare(
             actor=actor,
             system=system,