Add MCP gateway + authority introspection endpoint (0.4.0)

CHANGELOG.md

@@ -1,3 +1,179 @@
+## [0.4.0] - 2026-07-06
+
+### Added
+
+- **Authority dispatch path** — a hand-rolled JSON-RPC front-end for a first-party
+  server that authenticates tokens locally and serves its OWN tools (and, as a
+  gateway, aggregates + proxies upstreams) WITHOUT the official `mcp` SDK in the
+  request path. It coexists with the SDK-backed satellite path
+  (`McpToolkit::Server.build`) by design — the gem now carries two dispatch
+  front-ends, each for its role. The satellite path is unchanged.
+  - `McpToolkit::Protocol` — JSON-RPC constants + envelope helpers
+    (`SUPPORTED_VERSIONS`, `LATEST_VERSION`, `JSONRPC_VERSION`, `ErrorCodes`,
+    `Error` + subclasses with `#code`/`#data`/`#to_h`, `success_response` /
+    `error_response`). The byte contract of a first-party endpoint's error
+    envelope + version negotiation.
+  - `McpToolkit::Dispatcher` — `new(context:, config:)` + `#handle_request(request)`.
+    Dispatches `initialize` / `initialized` / `tools/list` / `tools/call` / `ping`
+    and the custom `notifications/<app>/tools/list_changed` cache-bust. `tools/list`
+    merges the host's own tool definitions with the gateway's namespaced upstream
+    tools; `tools/call` routes a namespaced name to `Gateway::Proxy` (translating
+    `UnknownUpstream` → method-not-found and relaying an upstream JSON-RPC error
+    verbatim) or a host tool (scope-gated) otherwise. Server identity + negotiated
+    versions come from config; NO SDK touchpoint.
+  - `McpToolkit::Authority::ControllerMethods` — the authority transport as an
+    includable concern. Every billing/tenancy step is an overridable hook
+    (`mcp_authenticate!`, `mcp_rate_limit!`, `mcp_track_usage`, `mcp_flush_usage`,
+    `mcp_resolve_account`, `mcp_session_data`, `mcp_dispatch`, `mcp_health_payload`,
+    `mcp_config`), each defaulting to a config callable so a PURE host needs no
+    subclass. The per-request loop RE-RESOLVES the account for every JSON-RPC call
+    — including each element of a batch — so a mixed-account batch still meters one
+    usage event per call against the right account (the batch is never delegated to
+    a bulk handler that couldn't re-resolve per element).
+  - `McpToolkit::Authority::ServerController` — a base controller (concern wired in,
+    lazily-parented) a host subclasses when its hooks touch app models (the
+    recommended path).
+  - `McpToolkit::Authority::Context` — the per-request context threaded into the
+    dispatcher + tools: `account`, `principal`, `bearer_token`, and a derived
+    `superuser?` (duck-typed off the principal).
+  - `McpToolkit::Tools::AuthorityBase` — an optional base for a host's own tools
+    (class DSL `tool_name` / `description` / `input_schema` /
+    `required_permissions_scope` / `definition`; `.call(context:, **arguments)`
+    entry; context accessors; `ensure_resource_accessible!`; ArgumentError →
+    InvalidParams / StandardError → InternalError mapping). Host tools plug in
+    through the api-agnostic `config.tool_provider` seam — the gem never references
+    a host's API layer, serializers, or resource catalog.
+- **`config.tool_provider`** — the api-agnostic tool seam. Duck-typed:
+  `provider.tool_definitions(context) -> [{ name:, description:, inputSchema: }]`
+  (context lets the host hide superuser-only tools) and `provider.find(name) -> a
+  tool object` (responding to `#required_permissions_scope` + `#call(context:,
+  **arguments)`). The dispatcher enforces the per-tool scope gate CENTRALLY.
+- **Registry-backed authority tools** — the authority-path counterpart to the
+  satellite's SDK tools, so a first-party server can serve the SAME four generic
+  read tools (`resources` / `resource_schema` / `get` / `list`) over
+  `config.registry` through the hand-rolled dispatcher, reusing the existing
+  `ListExecutor` / `GetExecutor` / `ResourceSchema` / `Serialization` /
+  `FieldSelection` / `Filtering` UNCHANGED.
+  - `McpToolkit::Authority::RegistryToolProvider.new(config:)` — a `tool_provider`
+    serving the four generic tools; `find(name)` returns a tool instance, and each
+    tool declares NO static scope (the per-resource scope is enforced dynamically
+    at call time). The satellite SDK tool path (`McpToolkit::Tools::*`,
+    `McpToolkit::Server`) is untouched — this is added alongside it.
+  - `McpToolkit::Authority::Tools::{Resources,ResourceSchema,Get,List}` — the four
+    thin tools. Each resolves the `resource` argument against the registry
+    (InvalidParams for unknown), gates a `superusers_only?` resource against
+    `context.superuser?` (REFUSE in get/list/resource_schema, HIDE in resources),
+    gates the resource's `required_scope_for` against the principal, and (get/list)
+    requires a resolved `context.account`. Returns a raw Hash for the dispatcher to
+    wrap — distinct by design from the satellite tools' `MCP::Tool::Response`.
+  - `McpToolkit::Authority::CompositeToolProvider.new(*providers)` — composes
+    several providers (e.g. the RegistryToolProvider + a host's bespoke tools)
+    behind one `config.tool_provider`: `tool_definitions` concatenates in order,
+    `find` returns the first match.
+  - `config.generic_tool_name_prefix` (default `""`) — namespaces the four generic
+    Registry-backed tools. When set (e.g. `"foo_"`) the provider advertises and
+    resolves them as `foo_resources` / `foo_resource_schema` / `foo_get` /
+    `foo_list`, letting a host keep stable, namespaced tool names for existing
+    clients; the empty default keeps the bare base names.
+- **`McpToolkit::Resource` generic seams** (all api-agnostic) — `superusers_only!`
+  / `superusers_only?` (authority tools honor it), `note(text)` + reader (surfaced
+  by `resource_schema`), and `filter(name, type:, description:, &applier)` +
+  `custom_filters` — a resource-specific filter whose block narrows the scoped
+  relation from a TOP-LEVEL request param, so a host can express a relational
+  filter the generic equality/operator allowlist can't. `ListExecutor` applies the
+  matching custom filters BEFORE the allowlist filters (its only change).
+- **`McpToolkit::ResourceSchema` enrichment** — each attribute now advertises the
+  filter `operators` it accepts (derived from `Filtering::OPERATORS_BY_TYPE`), and
+  the resource `note` is passed through, so a client can discover exactly which
+  `{ op:, value: }` conditions `list` will accept.
+- **Server-vs-gateway identity split** — `config.gateway_client_name` /
+  `gateway_client_version` (each defaulting to `server_name` / `server_version`).
+  `Gateway::Client`'s handshake `clientInfo` now reads the GATEWAY identity, so an
+  authority can advertise its own `server_name` to its callers while keeping its
+  upstream handshake byte-identical.
+- **`config.supported_protocol_versions`** (default
+  `McpToolkit::Protocol::SUPPORTED_VERSIONS`) — the version set the authority
+  dispatcher negotiates.
+- **`config.rate_limiter` / `usage_recorder` / `usage_flusher`** — the authority
+  transport's billing hooks as config callables (all default `nil` / no-op).
+- **Built-in rate limiting** — `McpToolkit::RateLimiter`, a fixed-window
+  per-principal counter backed by `config.cache_store`, plus
+  `config.rate_limit_max_requests` (Integer, default `nil` = OFF) and
+  `config.rate_limit_window` (seconds, default `3600`). When a cap is set, the
+  authority transport's `mcp_rate_limit!` counts each request, sets the
+  `X-RateLimit-Limit` / `X-RateLimit-Remaining` / `X-RateLimit-Reset` headers on
+  every capped response, and over the limit renders a JSON-RPC error (code
+  `-32029`) at HTTP `429` with a `Retry-After` header. Two new overridable hooks —
+  `mcp_rate_limit_max_requests` (default `config.rate_limit_max_requests`) and
+  `mcp_rate_limit_key` (default `mcp_principal.id`) — let a host keep the cap in
+  its own constant/model or bucket the counter differently. `config.rate_limiter`
+  remains as an escape hatch that fully replaces the built-in when set. A pure
+  host that sets no cap is unaffected.
+- **`config.superuser_resolver`** — an optional `->(principal) -> Boolean` making
+  superuser a first-class, OPTIONAL gem concept. `Authority::Context#superuser?`
+  calls it when set, else falls back to duck-typing `principal.superuser?` (false
+  when the principal doesn't respond to it). Together with the existing
+  `superusers_only!` resource flag and the authority tools' gating, this
+  formalizes superuser gating; the default (no resolver, no superuser-aware
+  principal) is "no superusers".
+- **Lazy `parent_controller` (Constraint B)** — the engine's `ServerController` /
+  `TokensController` and the authority `ServerController` are no longer eager-
+  loadable files; they are built from the CURRENT config by
+  `McpToolkit.build_engine_controllers!`, triggered lazily via `const_missing` and
+  reset on each reload by the engine's `config.to_prepare`. The parent is therefore
+  read only at build time — after the host's initializers/to_prepare — so a host's
+  whole MCP initializer can live in `to_prepare`. `TokensController#introspect`
+  behavior is preserved exactly.
+
+### Removed
+
+- The engine's `app/controllers/mcp_toolkit/{server_controller,tokens_controller}.rb`
+  files, replaced by the lazy builder above (their routes + behavior are unchanged).
+
+- **Gateway / upstream layer** (`McpToolkit::Gateway::*`) — the generic,
+  SDK-independent machinery a central app uses to aggregate *other* MCP servers
+  and proxy calls to them, previously an app-only concern. All app-specific values
+  (upstream URLs, account-selector meta key, logger, timeouts) are injected via
+  `McpToolkit::Configuration`; nothing in the layer names a deployment.
+  - `McpToolkit::Gateway::UpstreamRegistry` — a PER-CONFIG registry of upstream
+    servers (`Upstream = Data.define(:key, :url)` with `#name_for`), exposed as
+    `config.upstreams` and reset with a fresh config (test isolation for free).
+    API: `#register(key:, url:)` (blank url ignored), `#reset!`, `#all`, `#find`,
+    `#split_tool_name`. Config sugar: `config.register_upstream(key:, url:)`.
+  - `McpToolkit::Gateway::Client` — a minimal Streamable-HTTP MCP client
+    (`#tools_list`, `#tools_call`) with single-shot session-loss recovery (HTTP
+    404 / JSON-RPC `-32001`), SSE `data:` unwrapping, and content negotiation. Its
+    `Client::Error` (< `McpToolkit::Error`) carries `jsonrpc_error` / `http_status`
+    and references NO transport/protocol-error class — the consumer maps it. The
+    handshake `clientInfo` and protocol version come from config
+    (`DEFAULT_PROTOCOL_VERSION` falls back to the wrapped `mcp` SDK's latest).
+  - `McpToolkit::Gateway::Aggregator` — namespaces + caches (`config.cache_store`,
+    `config.upstream_list_ttl`) each upstream's tool list, pulled CONCURRENTLY via
+    concurrent-ruby (wrapped in `Rails.application.executor` when a booted Rails app
+    is present, plain futures otherwise). Only a non-empty pull is cached; a stale
+    empty is a miss (poisoned-cache self-heal); a failing upstream degrades (omit +
+    log) rather than breaking the list.
+  - `McpToolkit::Gateway::Proxy` — proxies a namespaced call, forwarding the
+    resolved `account_id` as `_meta[config.account_meta_key]`. An unknown key raises
+    `McpToolkit::Gateway::UnknownUpstream`; an upstream failure raises
+    `McpToolkit::Gateway::UpstreamCallError` (carrying `jsonrpc_error` /
+    `http_status`). Neither is mapped to a protocol-error class here.
+- **Authority introspection endpoint** — `McpToolkit::TokensController#introspect`,
+  drawn by the engine at `POST /mcp/tokens/introspect`, so a central app answers
+  introspection with no controller of its own. Its parent class is configurable via
+  `parent_controller` (like `ServerController`). Safe to draw unconditionally: with
+  no `token_authenticator` it answers `{ valid: false }`.
+- **`McpToolkit::Session#data`** — an opaque payload attachable at
+  `create!(data:)` and round-tripped through `find`, so an authority can bind a
+  session to a token id (letting a revoked token kill an in-flight session). The
+  gem does not interpret it; legacy rows default to `{}`.
+- `McpToolkit::Configuration` gains `upstreams` (a `Gateway::UpstreamRegistry`),
+  `register_upstream`, `upstream_timeout` (default `10`), `upstream_list_ttl`
+  (default `900`), and `logger` (default `nil`; all gateway/session call sites
+  guard with `logger&.`).
+- `concurrent-ruby` is now a direct dependency (already transitive via
+  activesupport) — the aggregator's parallel upstream pulls require it.
+
 ## [0.3.0] - 2026-07-03
 
 ### Added
@@ -119,5 +295,6 @@ hand-rolled controller wiring.
 
 ### Notes
 
-- The gateway / upstream-aggregation layer (`Mcp::Upstreams*`) is intentionally
-  out of scope (core-only).
+- The gateway / upstream-aggregation layer was intentionally out of scope for this
+  initial extraction. It was later extracted into the gem in 0.4.0 (see above), as
+  `McpToolkit::Gateway::*` — fully config-injected and app-agnostic.