feat: browser-native Globus Transfer + plugin detection

CHANGELOG.md

@@ -14,6 +14,48 @@ Pre-1.0, breaking changes bump the MINOR version.
   `npm ci` resolves it from the registry like any other package.
 
 ### Added
+- **Browser-native Globus Transfer** (#53, new `./transfer` subpath) — data
+  movement in and out of launched instances with no local machine and nothing
+  installed. Globus **Transfer** is not Globus Connect **Personal**: it moves data
+  between *managed* collections (an HPC DTN, an S3 collection) over REST, and every
+  endpoint is CORS-enabled. `transferClient()` covers `endpoint_search`,
+  `submit_transfer`, `task`/`task_list`, cancel, and an `awaitTask` polling loop —
+  all over an injected `fetch`, so it is unit-tested with no credentials.
+  - `submitTransfer` fetches a `submission_id` first: that is Globus's idempotency
+    mechanism, and skipping it lets a network retry move the data twice.
+  - An empty item list is refused — Globus would accept it and return a task that
+    succeeds having moved nothing, which reads as a silent failure.
+  - `GlobusTransferError` carries Globus's own `code`, and `needsConsent`
+    separates a fixable missing consent from a flat permission denial (both 403).
+  - `INACTIVE` is not terminal, and `niceStatus` is surfaced so a stuck task says
+    *why*.
+- **The Globus Transfer scope, opt-in** — `GlobusConfig.requestTransfer` adds
+  `TRANSFER_SCOPE` and `completeLogin` returns `GlobusTokens.transferToken` from
+  the *same* sign-in (Globus issues one token per resource server). Deliberately
+  **not** in `DEFAULT_SCOPE`: that would show every signing-in user a consent
+  screen about managing their transfers and reading their files, including users
+  who only want to see their instances. An absent `transferToken` is a normal
+  outcome, not an error.
+- **Plugin detection and launch-time declaration** (#53). Two columns, not one:
+  the browser can **declare 7 of 12** plugins at launch and **detect all 12**.
+  - `parsePluginTag` / `detectPlugins` / `instancePlugins` decode the
+    `spore:plugin:<name>` provenance tag. An absent or unparseable tag reads as
+    "unknown", never "not installed"; `verify=none` stays distinct from a missing
+    `verify=`; unknown `key=value` pairs are preserved rather than dropped.
+    Go's `"(none)"` digest placeholder is normalised away.
+  - `LaunchSpec.plugins` writes `/etc/spawn/plugins.json` in user-data, which
+    spored reads at startup — byte-compatible with Go's `plugin.Declaration`.
+    Written *before* spored starts, unlike the Go bootstrap, which appends it
+    after `systemctl start spored`; spored reads the file once at startup, so that
+    ordering is a race.
+  - `canDeclareAtLaunch` / `validateDeclarations` refuse the other five **with
+    distinct reasons**. Four need a locally-minted secret pushed to the instance
+    and would park at `StatusWaitingForPush` — a limitation of Go's own async path
+    (`pkg/pluginruntime/runtime.go:62`), not a browser gap. `spore-sync`'s local
+    half is mutagen on the developer's machine and stays the CLI's job.
+    Rejections are returned, not thrown, so a caller can launch with what works
+    and still say what was dropped.
+  - See `docs/data-movement.md`.
 - **Cross-account launch for the live smoke test** (#38) — the real-aws tier can
   now role-chain from the OIDC identity-anchor account into a separate compute
   account, so the ephemeral instance launches there rather than in the anchor

docs/README.md

@@ -14,6 +14,9 @@
 - **[API reference](api.md)** — the `SpawnClient` public API. The generated
   [TypeDoc reference](https://spore-host.github.io/spawn-ts/api/) is published
   alongside the demo.
+- **[Data movement and plugins](data-movement.md)** — browser-native Globus
+  Transfer (no local machine required), and the plugin split: 7 of 12 declarable
+  at launch, 12 of 12 detectable via the `spore:plugin:*` tag.
 - **[Integration with truffle-ts](integration.md)** — how the launcher/lifecycle
   (spawn-ts) and instance-discovery (truffle-ts) tools compose, and the
   tag-emit-vs-execution boundary.

docs/data-movement.md

@@ -0,0 +1,223 @@
+# Data movement and plugins from a browser
+
+A job is useless without a way to get data in and out. This document covers the
+two capabilities that make that possible without any locally installed software,
+and the one place where the browser deliberately stops.
+
+## The correction that made this possible
+
+Globus **Transfer** is not Globus Connect **Personal**.
+
+The `globus-personal-endpoint` plugin needs `globus whoami` on a laptop, and for a
+while that fact was read as "Globus needs a local machine". It does not. That
+plugin wraps Connect *Personal* specifically — the thing that turns a laptop into
+an endpoint. The Transfer API moves data between **managed collections** (an HPC
+DTN, an S3 collection, campus storage) over plain REST, and every endpoint is
+CORS-enabled. Preflighted live from `Origin: https://spore.host`:
+
+| endpoint | ACAO | allow-headers |
+|---|---|---|
+| `POST transfer.api.globus.org/v0.10/endpoint_search` | `*` | `authorization` |
+| `GET  transfer.api.globus.org/v0.10/task_list` | `*` | `authorization` |
+| `POST auth.globus.org/v2/oauth2/token` | `*` | `*` |
+
+So a browser can search collections, submit a transfer and poll it to completion
+with no local client and no proxy.
+
+## Getting a Transfer token
+
+We already sign in to Globus (`src/auth/globus.ts`, authorization-code + PKCE).
+Globus issues **one access token per resource server** and returns the extras in
+`other_tokens` — the same mechanism that already carries the OIDC `id_token`. So
+requesting the Transfer scope yields a Transfer token from the *same* sign-in:
+
+```ts
+import { beginLogin, completeLogin } from "@spore-host/spawn-ts/auth";
+
+await beginLogin({ clientId, redirectUri, requestTransfer: true });
+// …redirect back…
+const tokens = await completeLogin({ clientId, redirectUri });
+tokens.idToken;        // → AWS STS (AssumeRoleWithWebIdentity), as before
+tokens.transferToken;  // → the Transfer API
+```
+
+`requestTransfer` is **opt-in, not the default**. Putting `TRANSFER_SCOPE` into
+`DEFAULT_SCOPE` would show every signing-in user a consent screen asking to manage
+their transfers and read and write their files — including a user who only wants
+to look at their instances. Consent is asked for when data movement is actually on
+offer.
+
+`transferToken` being `undefined` is a **normal outcome**, not an error: a user can
+decline the Transfer consent and still sign in. Treat it as "data movement
+unavailable", not as a failure.
+
+## Moving data
+
+```ts
+import { transferClient } from "@spore-host/spawn-ts/transfer";
+
+const gt = transferClient({ transferToken: tokens.transferToken! });
+
+const [dtn] = await gt.searchCollections("ncsa#dtn");
+const task = await gt.submitTransfer({
+  sourceCollectionId: dtn.id,
+  destinationCollectionId: s3Collection.id,
+  label: "run-42 results",
+  items: [{ sourcePath: "/scratch/run-42/", destinationPath: "/out/run-42/", recursive: true }],
+});
+
+const final = await gt.awaitTask(task.taskId, {
+  onUpdate: (t) => console.log(t.status, t.filesTransferred, "/", t.filesTotal),
+});
+```
+
+`fetch` is injected throughout (the convention `completeLogin` already uses), so
+the whole client is unit-tested with no credentials and no network.
+
+Four details in that client are deliberate:
+
+- **`submitTransfer` fetches a `submission_id` first.** That extra round-trip is
+  Globus's idempotency mechanism: a retried POST carrying the same
+  `submission_id` is recognised as a duplicate. Skipping it lets a network retry
+  move the data **twice**.
+- **An empty `items` array is refused.** Globus accepts it and returns a task that
+  succeeds having moved nothing, which reads to a user as a silent failure.
+- **`verifyChecksum` defaults to `true`** (Globus's own default). Trading
+  integrity for speed is not this layer's call.
+- **An unrecognised task status maps to `ACTIVE`.** The direction of the guess
+  matters: guessing `SUCCEEDED` would report a transfer complete that never ran,
+  and guessing `FAILED` would invent a failure. `ACTIVE` only causes more polling.
+
+### Task states
+
+`INACTIVE` is **not terminal** — Globus resumes such a task once credentials are
+refreshed — so only `SUCCEEDED` and `FAILED` end the poll (`isTerminal`). When a
+task is stuck, `niceStatus` / `niceStatusDescription` carry the reason; a UI
+showing only `INACTIVE` leaves the user with nothing to act on.
+
+`awaitTask`'s timeout throws with the **last observed state** in the message and
+says the transfer is still running at Globus. Giving up on watching is not the same
+as the transfer stopping, and a bare "timed out" loses both facts.
+
+### Errors
+
+`GlobusTransferError` carries Globus's own `code`, not just the HTTP status,
+because the code holds more information: `ClientError.NotFound`,
+`PermissionDenied` and `ConsentRequired` are all 4xx and all need different
+handling. `err.needsConsent` is the one that's **fixable by re-authenticating** —
+the difference between a working "Grant access" button and a dead end.
+
+An unparseable 200 raises `MalformedResponse` rather than returning an empty list:
+a broken response must not be indistinguishable from "there is nothing here".
+
+### The other direction: `mountpoint-s3`
+
+The `mountpoint-s3` plugin mounts an S3 bucket as a filesystem on the instance,
+and it is remote-only — pure user-data, no local half. Between Globus Transfer and
+`mountpoint-s3`, data moves both ways with nothing installed locally.
+
+## Plugins: two columns, not one
+
+Go spawn has two plugin transports and only one needs a controller:
+
+1. **`spawn plugin install`** on a running instance POSTs to
+   `http://127.0.0.1:7777/v1/plugins/install` through an SSH tunnel. A browser has
+   no SSH. Not portable.
+2. **`spawn launch --plugin`** writes declarations to `/etc/spawn/plugins.json` in
+   user-data, which spored reads at startup. Pure user-data — a browser can do
+   this.
+
+Detection, by contrast, is **universal**: every installed plugin leaves a
+`spore:plugin:<name>` EC2 tag, and `DescribeInstances` already returns it. So the
+matrix has two columns, and carrying only one of them would wrongly read as "no
+plugin support in the browser":
+
+| plugin | declare at launch | detect |
+|---|---|---|
+| cloudwatch-agent, code-server, docker, jupyterlab, mountpoint-s3, rstudio-server, vscode-tunnel | **yes** | yes |
+| github-actions-runner, globus-personal-endpoint, rclone, tailscale | no | **yes** |
+| spore-sync | no | **yes** |
+
+**7 of 12 installable; 12 of 12 detectable.**
+
+### Why the other five are refused rather than attempted
+
+Four of them have a local half that **mints a secret and pushes it** to the
+instance (a runner registration token, an auth key, an rclone config, Globus
+credentials). The launch-time path has no controller to run that step, so those
+plugins park at `StatusWaitingForPush` and are never resumed — and this is a
+documented limitation of **Go's own** async path, not a browser gap
+(`pkg/pluginruntime/runtime.go:62`). Accepting them would produce an instance with
+a plugin stuck forever and nothing to explain why, which is worse than refusing.
+
+`spore-sync` is the fifth and pushes nothing, but its local half is mutagen running
+on the developer's own machine. That is legitimately the CLI's job and it stays
+there.
+
+`canDeclareAtLaunch` returns the **reason** alongside the boolean, and the reasons
+are distinct on purpose: "needs a pushed secret", "belongs to the CLI", and "not a
+known plugin" are three different problems, and only the last one might be a typo.
+
+```ts
+import { validateDeclarations } from "@spore-host/spawn-ts";
+
+const { accepted, rejected } = validateDeclarations([{ ref: "jupyterlab" }, { ref: "tailscale" }]);
+// accepted: [{ ref: "jupyterlab" }]
+// rejected: [{ ref: "tailscale", reason: "…would park at waiting-for-push…" }]
+await client.launch({ ...spec, plugins: accepted });
+```
+
+Rejections are **returned, not thrown**, so a caller can launch with what works
+while telling the user exactly what was dropped. Silently filtering would produce
+an instance missing a plugin the user asked for, with nothing to explain it.
+
+### Reading what's deployed
+
+```ts
+import { instancePlugins, describePluginState } from "@spore-host/spawn-ts";
+
+const found = instancePlugins(instance);
+// [{ name: "spore-sync", version: "v1.2", contentSha256: "abc123def456",
+//    verify: "signature", parsed: true, raw: "…" }]
+```
+
+Three things `parsePluginTag` gets right, each a place where a simpler parser
+would assert something the data doesn't support:
+
+- **An absent or unparseable tag reads as "unknown", never "not installed".** The
+  tag is written best-effort (`recordPluginProvenanceTag` returns early when the
+  instance isn't EC2-resolvable and only warns on a tag-write failure), and it is
+  never written at all for launch-time declarations. So absence genuinely means
+  "we don't know" — hence `describePluginState([])` says so in words. An
+  unparseable *value* still yields a record with `parsed: false`, because the
+  tag's existence is itself the evidence that the plugin is deployed.
+- **`verify=none` is distinct from a missing `verify=`.** The first says "the
+  install ran and verification reached neither a signature nor a manifest" — a
+  supply-chain finding. The second says "we can't tell". Collapsing them hides the
+  first.
+- **Unknown `key=value` pairs are preserved in `extra`.** The Go builder can grow
+  fields, and a strict parser would silently lose provenance on newer instances.
+
+Go's `shortHash` writes the literal string `"(none)"` for an empty digest. That's
+a display placeholder, not a hash, so it is normalised to `undefined` — otherwise
+a UI would render `sha256=(none)` and imply a digest was recorded.
+
+## What still needs the CLI
+
+- `spawn plugin install` on a running instance (SSH tunnel to the on-instance
+  controller).
+- The local half of `spore-sync`, `rclone`, `tailscale`, `github-actions-runner`,
+  and `globus-personal-endpoint`.
+- Globus Connect **Personal** as an endpoint — though note the browser *can*
+  submit a transfer to a GCP collection; it will simply sit queued until the
+  user's machine is on. `TransferCollection.isGlobusConnectPersonal` is surfaced
+  so a UI can say that before the user waits an hour.
+
+## Live checks
+
+Both live paths are **manual and opt-in**, never CI:
+
+- Globus Transfer moves real data between real collections. Not something a test
+  run should trigger.
+- Plugin declaration at launch requires a real instance; the gated smoke test
+  (`docs/live-smoke.md`) is the place for it.

docs/integration.md

@@ -44,20 +44,59 @@ and does the work. Three tiers:
 | **Tag-emit only** | pre-stop, spot-interruption webhook, notify, active-processes | writes the tag; `spored` executes it on the box |
 | **Not portable** | FSx provisioning, DCV, on-node storage mounts, `logs`/`collect` | out of scope (needs the daemon / a backend) |
 
-This is why a spawn-ts launch is **wire-compatible** with the Go tool: the tags
-it writes are exactly what `spored` (and `spawn list`) expect, even for behaviors
-spawn-ts can't run itself. The docs are careful to say which tier a feature is in
-so nothing over-promises.
+This is why a spawn-ts launch is **wire-compatible** with the Go tool for the tags
+it does write: they are byte-for-byte what `spored` (and `spawn list`) expect, even
+for behaviors spawn-ts can't run itself. The docs are careful to say which tier a
+feature is in so nothing over-promises.
+
+**Wire-compatible is not yet wire-complete.** Go's `buildLaunchTags`
+(`spawn/pkg/aws/tags.go:32`) stamps 55 tags at launch; spawn-ts stamps 33. The
+absent ones are not all tier D — the base-identity block is tier **B**, and its
+absence has real consumers:
+
+| absent tag | tier | consequence |
+|---|---|---|
+| `spawn:iam-user` | **B** | the portal can neither list nor terminate a spawn-ts instance (`lambda/dashboard-api/instances.go:285` → 403); `cleanup --only-mine` skips it |
+| `spawn:account-base36` | **B** | `spored`'s notifier can't build the instance FQDN |
+| `spawn:os`, `spawn:local-username` | **B** | `spawn connect` can't infer the SSH user |
+| `spawn:version` | **B** | AMI management can't tell which launcher wrote the instance |
+| `spawn:active-ports` | **C** | `spored` writes this one itself; not a launcher gap |
+| FSx / EFS / DCV tags | **D** | the provisioning they describe isn't browser-reachable |
+
+Tracked in [#51](https://github.com/spore-host/spawn-ts/issues/51); the full
+50-command tier matrix is [#57](https://github.com/spore-host/spawn-ts/issues/57).
 
 ## Why the catalog is offline (and where live data goes)
 
 truffle-ts ships a bundled instance/price snapshot so `find` works with **zero
 credentials and zero cost** — the same cost-safe, MockProvider-default ethos as
 spawn-ts. Live AWS data (real-time `DescribeInstanceTypes`, spot prices, quotas)
-needs credentials a browser can't safely hold and hits CORS, so it lives behind
-truffle-ts's `Finder` seam (a Node/backend implementation), not in the default
-browser path. See truffle-ts's [catalog](https://github.com/spore-host/truffle-ts/blob/main/docs/catalog.md)
-and [architecture](https://github.com/spore-host/truffle-ts/blob/main/docs/architecture.md) docs.
+lives behind truffle-ts's `Finder` seam so the default path stays offline and
+free, **not because a browser can't reach it**.
+
+Two claims that used to appear here were wrong, and the distinction matters
+because it decides which features are portable at all:
+
+- **CORS is not a blocker.** Preflighted live from `Origin: https://spore.host`,
+  every endpoint the live finder needs returns `access-control-allow-origin: *`
+  with `access-control-allow-methods: POST` and
+  `access-control-allow-headers: content-type,x-amz-target,authorization`:
+  `ec2.us-east-1.amazonaws.com`, `api.pricing.us-east-1.amazonaws.com`,
+  `servicequotas.us-east-1.amazonaws.com`. So do `sts`, `ssm`, `tagging`, `ce`,
+  `dynamodb`, `scheduler`, and `bedrock-runtime`.
+- **A browser can hold usable credentials.** Not a long-lived key — a
+  short-lived STS session. `credsFromIdToken` ([aws-federation.ts](../src/auth/aws-federation.ts))
+  exchanges a Globus OIDC `id_token` for `AssumeRoleWithWebIdentity` credentials
+  that live in the tab and expire on their own. That is the same BYOA path the
+  portal already signs in with.
+
+What *is* still true: the live finder costs API calls (and `pricing:GetProducts`
+for savings annotations), needs IAM permissions the bundled path doesn't, and adds
+latency. Those are the reasons `BundledFinder` remains the default — a deliberate
+cost-safety choice, not a technical wall. See truffle-ts's
+[catalog](https://github.com/spore-host/truffle-ts/blob/main/docs/catalog.md) and
+[architecture](https://github.com/spore-host/truffle-ts/blob/main/docs/architecture.md)
+docs (both carry the same stale framing — truffle-ts#35).
 
 ## The spored relationship, in one line
 

package.json

@@ -31,6 +31,7 @@
     "./terminal": { "types": "./dist/terminal.d.ts", "import": "./dist/terminal.js" },
     "./quotas": { "types": "./dist/quotas.d.ts", "import": "./dist/quotas.js" },
     "./dns": { "types": "./dist/dns/index.d.ts", "import": "./dist/dns/index.js" },
+    "./transfer": { "types": "./dist/transfer/index.d.ts", "import": "./dist/transfer/index.js" },
     "./portal": { "types": "./dist/portal/portal-core.d.ts", "import": "./dist/portal/portal-core.js" },
     "./ui": { "types": "./dist/ui/index.d.ts", "import": "./dist/ui/index.js" },
     "./ui/style.css": "./dist/ui/dashboard.css"