diff --git a/CONTEXT.md b/CONTEXT.md index e19f45c7..fd4b209f 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -34,7 +34,11 @@ S15/S16 target terms below refine product intent. Definitions explicitly marked **Default standing-policy proposal (S15, POLICY15-02):** The first standing policy Jit offers on an installation with no configured policy. It is derived from what the installation actually has, with the product-default gates and a finite, renewable expiry. It is only a proposal: the owner's one exact press installs it, and a changed resource set produces a new proposal instead of changing installed rules. See ADR 0073. -**GitHub connection (S15, GIT15-04):** The owner's Web-pasted fine-grained token, held only in the protected vault, plus the revisioned list of repositories the owner selected for Jit. Selected repositories become team resources and default-proposal rows. The connection grants no publication; the product-default and personal GitHub profiles decide that. +**GitHub connection (S15, GIT15-04):** The owner's Web-pasted fine-grained token, held only in the protected vault, plus the revisioned list of repositories the owner selected for Jit. Selected repositories become team resources and default-proposal rows. The connection grants no publication; the product-default and personal GitHub profiles decide that. In S16 (ADR 0089) the owner connects, reconnects and revokes it on the client's Connections page, and a selection or removal updates the standing rules through one connection-membership revision. + +**Tool source (S16):** An owner-added HTTPS MCP server, OpenAPI service or GraphQL endpoint with owner-written operations (ADR 0089). Its discovered tools are data until the owner's one-time **tool review** decides, per tool, *reads only*, *changes data* (the Bot runs it), *needs my approval each time* or *off*; a server's hints are only pre-selected suggestions and a GraphQL mutation can never be a read. A changed catalog keeps the source connected but every tool waits for a new review. A source is a team resource every Bot may use unless the Bot's settings exclude it. + +**Connection-membership revision (S16):** The standing-policy revision Jit installs after a tool review, a GitHub selection or a removal, only when the installed rules differ from the default proposal in nothing but connection membership. Work that does not use the changed connection stays current (ADR 0089); any other difference waits for the owner on the rules page. **Persona:** Owner-authored configuration of address, language and tone. It is trusted configuration, not model-authored memory. @@ -162,7 +166,7 @@ The opt-in S15 TEAM/MEM implementation now retains relevant sourced subsets auto **Attachment:** An owner-supplied file sanitized out of process and always treated as untrusted data. Its contents cannot configure tools, routes or grants. -**MCP proposal:** A revisioned source/version/command or endpoint proposal accepted by the owner. An advertised tool or readOnlyHint does not itself grant access. +**MCP proposal:** A revisioned source/version/command or endpoint proposal accepted by the owner. An advertised tool or readOnlyHint does not itself grant access. In S16 an HTTPS MCP server is a tool source reviewed through the tool review; its sessions are per run and never shared (ADR 0089). **Local API:** An owner-enabled loopback API/CLI over the same authorized commands. It is not a general host shell. diff --git a/docs/adr/0089-s16-connections-and-tool-sources.md b/docs/adr/0089-s16-connections-and-tool-sources.md new file mode 100644 index 00000000..a25053c3 --- /dev/null +++ b/docs/adr/0089-s16-connections-and-tool-sources.md @@ -0,0 +1,52 @@ +--- +status: implemented for CONN16-01 (#417), MCP16-01 (#418) and OPEN16-01 (#419); deterministic and real-bundle browser evidence; live account evidence separate (#431) +date: 2026-11-08 +scope: CONN16-01 (#417), MCP16-01 (#418), OPEN16-01 (#419); S16-AC11/AC12; FR-10/15/17/21/22/23/35/37/38/39; NFR-02/05/06/07/09/14 +--- + +# Connections: GitHub and reviewed HTTPS tool sources in the client + +## Context + +S16-AC11 asks the owner to connect, reconnect and revoke a real app from the client (and from a conversation) without editing configuration, reuse it across authorized Bots and act through it under the same protected policy. S16-AC12 asks for an HTTPS MCP source and an OpenAPI/GraphQL or gateway source with versioned schemas, used under current purpose, recipient and action policy, with run-scoped sessions, reconciliation of remote uncertainty, and no authority from discovery or hints. + +Before this change GitHub was connected on the S15 `/resources/github` page (fine-grained token, GIT15-04) and its repositories reached the standing rules only after the owner re-installed the default proposal on `/rules` and registered them in TEAM. There was no way to add an MCP or API source, and POLICY treated every newer standing-policy revision as `stale_authority` for running work unless it differed only in other Bots (ADR 0088). + +Owner decisions on 2026-11-08, before this work: one PR for #417, #418 and #419 with one shared review/drift/reconcile contract; the first real app is GitHub with a fine-grained token (no OAuth app to create), moved into the S16 client with a chat entry, while Google stays on the S14 broker; every connection is a team resource that all Bots may use by default, with per-Bot removal in Bot settings; MCP/OpenAPI/GraphQL sources support no auth, a pasted bearer token or custom header, and MCP OAuth 2.1 with dynamic client registration, PKCE and state; a server's hints are only labelled suggestions and the owner confirms the discovered tool list once, choosing per tool between *reads only*, *changes data, the Bot runs it*, *needs my approval each time* and *off*, with anything not confirmed as a read suggested for approval and any schema/scope change needing a new review; in chat a Bot shows a card that opens the protected form, secrets are never typed into chat, and the owner returns to the same conversation. + +Reuse: the pinned Rakazo sources were compared. Its connector stack is a hosted catalog/OAuth broker behind its own runtime and database; adopting it would import a competing authority and credential store. No upstream source, dependency or asset was imported for this work; the existing S14 stdio MCP client informed the bounds of the new HTTP client. + +## Decision + +1. **Explicit opt-in.** `s15-runtime/v1` gains `connections: true` (composition `connections: { vault }`, requiring DATA15), set by the generated installer. It adds the additive store `connections.sqlite`. Secrets live only in the protected `ConnectionVault`. +2. **Connection directory** (`s16-connections/v1`, `src/s16/connections/service.ts`). A source record holds its kind (`mcp`, `openapi`, `graphql`), label, exact endpoint, auth facts (kind, header name, the verified secret's fingerprint, OAuth server and client identity), state, the latest catalog, the owner's review and the owner's document (OpenAPI text or GraphQL operations). It owns four identities, all derived from the adding request: the TEAM/POLICY resource `src_<24 hex>`, the DATA15 service recipient `srcr_…`, the private DATA15 source `srcs_…` and the catalog fence `srcc_…`. Owner commands (`s16-connection-command/v1`): `source.add`, `source.review`, `source.refresh`, `source.reconnect`, `source.document`, `source.oauth_begin`, `source.oauth_complete`, `source.remove`, `github.connect`, `github.select`, `github.remove` and `request.dismiss`, idempotent per `requestId` (the receipt identity binds a secret's or document's digest, never its text) and serialized. +3. **Protected network.** Every source request (MCP, OpenAPI, GraphQL, OAuth metadata, registration and token) uses the protected browser's checked egress: one DNS resolution, every answer must be a public address (loopback, private, link-local, metadata, unique-local, mapped and documentation addresses refuse `source_address_not_public`), the socket dials exactly the checked address, redirects refuse `source_redirect_refused`, responses are byte-bounded, and endpoints must be HTTPS without credentials or a fragment. +4. **Discovery is data.** MCP sources are discovered over the Streamable HTTP transport (spec 2025-06-18: `initialize`, `tools/list` within 16 pages and 128 tools). OpenAPI 3.0/3.1 documents (JSON or alias-free YAML) are imported by a bounded importer: one tool per operation with a fixed method, path template, path/query parameters and optional JSON body; only local `#/…` references are inlined (at most 512, cycles refuse); an operation with an external reference, a required header/cookie parameter, a parameter whose encoding is not the default (path `simple` scalars; query `form` scalars or scalar arrays, repeated or comma-joined as `explode` says) or a required non-JSON body is skipped and named. GraphQL sources are owner-written named operations (a persisted-operation catalog, no introspection): a trusted parser reads each operation's real type, variables and used fragments and bounds its expanded depth (12) and fields (256); subscriptions, anonymous operations and type definitions are unsupported; variable schemas keep nullability at every level. Schemas are bounded (16 KB, CHAT's tool bound) and `$ref`-free. A catalog's hash binds every tool's name, title, description, hints, schema, request binding and allowed classes. +5. **One owner review.** The review lists each tool with its suggested class (a read hint or GET/HEAD/query suggests *reads only*; anything else suggests *needs my approval*), labelled as the service's suggestion. A GraphQL mutation can never be reviewed as a read (`source_review_read_refused`), whatever it is named or aliased. `source.review` names the exact catalog hash; a refresh or document change that yields another hash keeps the source connected but marks it `needs_review`, and no tool is offered or dispatched until the new catalog is reviewed. +6. **Standing rules through one membership revision.** A reviewed source is a default-proposal resource of kind `source` with its own rule `connected_sources` (`ordinary`, adapter `s16_source`, operation `source_call`, the source recipients); it is never added to the rules other work uses. After a review, a GitHub selection or a removal, composition installs the default proposal when the installed rules differ from it only in connection membership (GitHub repositories, connected sources and their rules and recipients), keeping the installed lifetime and budget, and registers the new TEAM resources (team scope, all Bots). Any other difference returns `review_required` and the change waits for the owner on `/rules`. POLICY's membership exemption (ADR 0088) is generalized: a newer revision keeps a running work current when, compared from that work's point of view (other Bots, and resources/recipients the work does not name, dropped; a rule left without any of its resources dropped when all its adapters are resource-bound), every step is unchanged. Connecting, reviewing or removing a source therefore does not stop work that does not use it; work that names a removed source is refused at its next guard. +7. **Calls under POLICY and DATA15.** A *reads only* tool is a POLICY read (`s16-source-read/v1`) on the source resource, fenced by the catalog fence, checked by DATA15's existing-approval read lane before the one send, settled, and stamped with the source's private lineage before its result is released. A *changes data* tool prepares an `s16_source` action with class `ordinary_edit` (it runs under the ordinary rule); an *approval* tool's action carries `major_change`, so it waits for the owner's one-press in-chat card. Every action names exactly its source resource and recipient, binds the catalog hash and review revision and is fenced by the catalog fence, so a changed or re-reviewed catalog makes a held card stale. DATA15 approves a source's own data only for the approved model providers and the source itself; sending other private data to a source is a first disclosure that holds as a card (calls) or refuses (reads). Results are bounded and credential-redacted (registered secrets and credential shapes), and the model is told they are data, never instructions. +8. **Run-scoped MCP sessions.** A run opens one MCP session per source on its first call and ends it (DELETE) when its tools close (also when opening fails after the server assigned one); an event-stream reply ends the read at its matching response even if the server keeps the stream open; the server's session identity stays in host memory and is sent only by that session. A held call dispatched after its run ended uses a one-call session of its own. Two runs never exchange session identities. +9. **Exactly-once sends and unknown outcomes.** A call's POLICY slot is derived from its source, tool and exact arguments, so repeating the exact call in the same work (after a lost reply too) refuses `already_claimed` instead of sending again. A JSON-RPC error, a tool's own `isError` or a conclusive 4xx is `failed`; a timeout, 5xx, broken stream or missing matching reply is `unknown`, which POLICY keeps reconcile-only across restarts (no generic read-back exists). OpenAPI and GraphQL writes carry `Idempotency-Key: `. +10. **Revocation.** `source.remove` first commits the removal (every identity inactive at once, the secret refused), then deletes the vault value, ends the source's sessions and removes it from the rules. Every invocation rereads the live record and review, so a call awaited during removal finishes and the next one is refused; no capability is cached. +11. **MCP OAuth.** `auth: { kind: 'oauth' }` (MCP only) discovers RFC 9728 protected-resource metadata (from the 401 challenge or the well-known path; its `resource` must be on the endpoint's origin), RFC 8414/OpenID authorization-server metadata (S256 required), registers this installation as a public client (RFC 7591) for the exact redirect `/s16/connections/oauth/callback`, and returns an authorization URL with PKCE S256, a single-use 256-bit `state` and the RFC 8707 `resource`. The redirect carries no session (the owner cookie is SameSite=Strict): the state alone, single-use, bound to the source revision and owner generation and valid for ten minutes, stores the code write-only; a wrong, reused, expired or superseded state refuses and stores nothing. The owner's authenticated `source.oauth_complete` exchanges the code once with the verifier and discovers with the token; tokens are fingerprint-registered and refreshed shortly before expiry, one refresh per source at a time (rotating refresh tokens); a refresh result is saved only if the source, its auth and the stored token are unchanged since it started, and a refused refresh asks the owner to sign in again. +12. **GitHub in the client.** The GIT15-04 connection is exposed as the GitHub app: `github.connect` (fine-grained token, write-only), `github.select` and `github.remove`, followed by the same membership revision. Reconnecting with a new token keeps the repositories it can still reach; an expired token shows *sign-in needed*. Publication, review replies and merge after required checks run under the AUTH16-01 product default; production deployment still waits. +13. **Chat entry.** Any non-delegated, non-file work gets `propose_connection` (`github`, `mcp`, `openapi`, `graphql`, optional HTTPS endpoint and label). It stores an inert card for the request; an endpoint with credentials or a query string refuses. Pressing the card opens the Connections page prefilled, with a way back to the conversation; connecting approves nothing the conversation is waiting for. +14. **Per-Bot availability.** A source is a team resource for every Bot. `bot.update` may carry the Bot's exact `excluded` team resources (connections included); the composed catalog refuses an excluded resource for that Bot at admission and dispatch, and the Connections page lists which Bots may use each source. +15. **Bounded runs and recovery.** A run offers at most 64 tools (CHAT's bound): the run's own tools first, then connected-source tools in source and catalog order; the rest are not offered to that run. Model-facing tool names that would exceed 64 characters keep a digest of the full name. A command's first source change records it as pending, so a retry with the same request ID after a stop finishes it (rediscovers, or reruns the rule update) instead of conflicting. +16. **Owner surface.** `GET /api/s16/client/connections` (`s16-client-connections/v1`) and `POST /api/s16/client/connections/commands` (a body of at most 1 MiB plus envelope, owner and CSRF rechecked after the body). Capabilities gain `connections`. The React client adds the Connections page (`/s16/connections`, tabs *Apps* and *Tool sources*, add/reconnect forms, the review panel, state, Bot availability and rule status), the chat card, a *Use this connected tool?* approval card kind, and Bot settings for excluded resources, in English and Thai. The rules page lists connected sources. + +## Consequences + +- A source's tools are as safe as the owner's review: a tool marked *changes data* runs without a card. Defaults lean to approval, and a changed catalog always returns to review. +- A work that already read other private data (for example a private repository) cannot read from a source until that data may go to it; a call holds as a first-disclosure card instead. +- Unknown outcomes have no automatic read-back; the owner records what happened (S15 reconcile-only statements). +- MCP server-to-client features (sampling, roots, elicitation), the standalone GET stream and `tools/list_changed` notifications are not used; the owner refreshes a source explicitly. OAuth servers without dynamic registration are unsupported for now (the page shows the redirect URI). OpenAPI header/cookie parameters and non-JSON bodies are unsupported. +- Owner-customized standing rules do not follow connection changes automatically; the rules page shows the update. +- Live evidence (a real hosted MCP server, a real OAuth sign-in, a real OpenAPI/GraphQL service, a real GitHub account) belongs to LIVE16-01 (#431). + +## Alternatives considered + +- *Trust MCP `readOnlyHint` / HTTP method*: zero configuration, but a server could relabel a destructive tool; rejected (S16-F12). +- *GraphQL introspection with model-written queries*: flexible, but a model-written document could alias a mutation and is unbounded; owner-written operations bind the exact document. +- *Add sources to the main ordinary rule*: fewer rules, but every connection change would make all running work stale; a dedicated rule plus the generalized membership exemption keeps unrelated work current. +- *A per-connection approval of the policy on /rules*: explicit, but one more press with no new information after the tool review; the membership revision installs only what the review already named and refuses any other difference. diff --git a/docs/contracts/s16-connections-v1.md b/docs/contracts/s16-connections-v1.md new file mode 100644 index 00000000..08d52567 --- /dev/null +++ b/docs/contracts/s16-connections-v1.md @@ -0,0 +1,63 @@ +# s16-connections/v1 + +CONN16-01 (#417), MCP16-01 (#418) and OPEN16-01 (#419). Decision record: [ADR 0089](../adr/0089-s16-connections-and-tool-sources.md). Exported TSDoc in `src/s16/connections/` (`types.ts`, `service.ts`, `runtime.ts`, `mcp.ts`, `oauth.ts`, `openapi.ts`, `graphql.ts`, `catalog.ts`, `schema.ts`, `net.ts`, `disclosure.ts`, `tools.ts`), `src/s15/policy/service.ts` (`sameForWork`, `resourceBoundAdapters`), `src/s15/policy/defaults.ts` (`connected_sources`) and `src/s15/composition.ts` (`connectionMembership`) is the executable contract; this page summarizes it. + +## Selection + +- Runtime config `s15-runtime/v1` field `connections: boolean` (absent: off; `true` requires `disclosure`). The generated S15 installer config sets `true`. +- Composition option `connections: { vault, network? }` adds the additive store `connections.sqlite`. `network` (`allowHttp`, `resolve`, `connect`) is a deterministic test seam only; the public-address rule is not a seam. +- Off means: no connection routes, no `propose_connection` tool and `capabilities.connections = { available: false, code: "connections_unavailable" }`; GitHub stays on `/resources/github`. + +## Protected state (`connections.sqlite`, application id `0x4a494337`) + +| Table | Meaning | +| --- | --- | +| `sources(id PK, record_json)` | `s16-source/v1` records. States `authorizing`, `needs_review`, `active`, `reconnect_required`, `unavailable`, `removed` (terminal). | +| `flows(id PK, state_hash UNIQUE, record_json)` | OAuth flows `pending` → `received` → `exchanging` → `complete` / `failed`; only the newest flow of a source can complete. | +| `requests(id PK, record_json)` | Chat connection cards (`open`, `done`, `dismissed`). At most 64 open. | +| `commands(id PK, payload_hash, result_json)` | Owner command idempotency; the hash binds a secret's or document's digest. A stored OAuth receipt omits the sign-in URL. | +| `pending(id PK, payload_hash, source_id)` | A command whose first source change committed but whose receipt did not; the same request finishes it. | + +Vault keys (installation-scoped aliases): `source_` holds a bearer/header secret or the OAuth token set; `source_flow_` holds a flow's state, PKCE verifier, client secret and, after the callback, the code. + +Bounds: 32 sources, 128 tools per source (16 list pages), documents and responses 1 MiB, results and arguments 64 KiB, schemas 16 000 bytes and depth 12, at most 64 tools offered per run, descriptions 1024 characters, request 30 s, call 120 s, OAuth flow 10 minutes, GraphQL depth 12 and 256 fields, 512 OpenAPI references. + +## Identities + +For a source `src_` (24 hex from the adding request): resource `src_`, DATA15 recipient `srcr_`, DATA15 private source `srcs_` (always resolvable, never declassified) and catalog fence `srcc_` (kind `input`, revision = review revision). Resource and recipient are active while the source is reviewed and not removed; the fence is active only while the review covers the current catalog and the source is `active`. + +## Owner API + +- `GET /api/s16/client/connections` → `s16-client-connections/v1`: `github` (`state`, `login`, expiry, accessible/selected repositories, `policy`), `sources` (`SourceView`: state, code, auth kind/header/issuer, catalog and reviewed hashes, `changed`, tools with `hint`, `suggested`, `allowed`, `decision` and request summary, skipped entries, Bots and whether each may use it, `policy`, `origin` conversation), open chat `requests`, `limits`, `oauthRedirect`. Never a secret, token, code or session identity. +- `POST /api/s16/client/connections/commands` (`s16-connection-command/v1`) → `s16-connection-receipt/v1` (`source`, `github`, `authorize` for OAuth, `policy`: `installed` / `unchanged` / `review_required`). + +| Command | Effect | +| --- | --- | +| `source.add` | `sourceKind`, `label`, `endpoint` (empty for an OpenAPI document's own server), `auth` (`none`, `bearer {secret}`, `header {name, secret}`, `oauth`), `document` (OpenAPI/GraphQL), `origin` (chat card). Stores the secret write-only, then discovers; OAuth returns `authorize`. | +| `source.review` | `catalogHash` and a decision for every tool (`read`, `change`, `approval`, `off`; a mutation never `read`). Installs the membership revision. | +| `source.refresh` / `source.document` | Rediscover / replace the document; a new catalog hash means `needs_review`. | +| `source.reconnect` | New auth; ends the source's sessions; rediscovers or begins OAuth. | +| `source.oauth_begin` / `source.oauth_complete` | New sign-in URL / one-time code exchange, then discovery. | +| `source.remove` | Identities inactive at once, vault value deleted, sessions ended, membership revision. | +| `github.connect` / `github.select` / `github.remove` | GIT15-04 connection; selection and removal install the membership revision. | +| `request.dismiss` | Hide a chat card. | + +- `GET /s16/connections/oauth/callback?state&code|error` (no session): single-use state; 303 to `/s16/connections#oauth=` or `#oauth_error=source_oauth_stale`. +- `bot.update` (`s16-bot-command/v1`) may carry `excluded` (exact team resource IDs) — additive. + +## Work surface + +- Tool names `