feat(go-validate): Go fast-path sidecar for the /validate hot path - #1653
Open
aarora79 wants to merge 27 commits into
Open
feat(go-validate): Go fast-path sidecar for the /validate hot path#1653aarora79 wants to merge 27 commits into
aarora79 wants to merge 27 commits into
Conversation
Adds a stdlib-only Go sidecar that fronts the auth-server's /validate auth_request endpoint. It verifies the configured-IdP RS256 bearer path (cached JWKS, atomic keyset, last-good retention), maps claims to identity, mints the HS256 X-Internal-Token-Registry, and writes the identity headers nginx consumes. Everything it does not recognize (cookies, other IdPs, opaque tokens, unknown kid, or an unconfigured fast path) is reverse-proxied to the unchanged Python auth-server, so it is byte-identical where it answers and zero-risk where it does not. Security invariants baked in: strips client-supplied identity/trust headers before minting; validates SECRET_KEY for missing/weak at startup (fail closed); enforces iss/aud from config, never from the token; recognized-invalid -> 401, unrecognized -> fallback (never a silent allow). Refs #1652
…runs it
Makes the fast path self-contained: build_and_run.sh now brings up the
go-validate sidecar and routes the nginx /validate auth_request through it,
with no separate step.
- nginx templates: give /validate its own {{VALIDATE_UPSTREAM_HOST/PORT}}
placeholder (oauth2/* stay on the auth-server).
- nginx_service.py: resolve /validate upstream from VALIDATE_UPSTREAM_URL,
defaulting to the auth-server when unset (backward compatible for
Terraform/ECS, Helm, and podman/prebuilt compose).
- docker-compose.yml: add the go-validate service (build, env, healthcheck
via a new -healthcheck self-check flag) and point the registry's
VALIDATE_UPSTREAM_URL at it.
- .env.example: document VALIDATE_UPSTREAM_URL / VALIDATE_JWKS_URL /
VALIDATE_ISSUER / VALIDATE_AUDIENCE / JWKS_REFRESH_SECONDS.
- go-validate: audience mismatch now falls back to Python (authoritative)
instead of 401, so inline-by-default never rejects a token Python would
accept; only bad signature/expiry -> 401.
- tests: cover the VALIDATE_UPSTREAM_URL set + default-to-auth-server paths.
Refs #1652
…o Python Two correctness fixes found during live testing behind nginx: - Fallback must not mutate the request. The handler previously stripped identity headers (incl. X-Client-Id) before every fallback, but nginx sets X-Client-Id / X-Original-URL / X-Registry-Api-Auth as legitimate INPUTS on the /validate subrequest. Stripping them made the fallback diverge from a direct nginx->Python call and broke POST /api/tokens/generate (401). Fallback is now byte-identical; identity injection on the fast path is prevented by Set-ting the response headers, not by mutating the request. - Audience mismatch now defers to Python (fallback) instead of 401, so the fast path never rejects a token the full handler would accept. Refs #1652
The fast path now produces byte-identical /validate output to Python for the RS256 bearer path, verified live against auth-server: - Scope resolution: load mcp_scopes + idp_m2m_clients into TTL-refreshed in-memory snapshots (Mongo driver) and resolve X-Scopes exactly as Python's map_groups_to_scopes + M2M group enrichment, honoring the enabled flag (fail closed) and the user-generated sentinel. Falls back to Python for cases it cannot replicate (e.g. user tokens needing idp_user_groups). - Per-hop internal tokens: mint the registry-ui token (thin identity, no scopes) only when X-Registry-Api-Auth is set, and the mcp-proxy token (scopes + resolved upstream, gated on the source-secret marker) only when X-Resolved-Upstream is set - matching internal_request_token.py audiences, claim shapes, and canonical auth_method (per-user IdP -> oauth2). - Headers: X-User/Username/Client-Id/Scopes/Auth-Method/Server-Name/Tool-Name/ Groups all match; verified byte-identical on the /api/ path. Adds the official Go MongoDB driver (read-only snapshots). Compose wires the DOCUMENTDB_* env into the sidecar. Refs #1652
…uest nginx's auth_request subrequest for a POST/PUT/PATCH origin forwards the original Content-Length with NO body. httputil.ReverseProxy then blocked copying that phantom body to the auth-server, hanging the subrequest until nginx timed out (504 -> auth_request collapsed to 500). Every mutating request through the gateway (e.g. POST /api/tokens/generate) failed; GETs were unaffected. Python never hit this because uvicorn does not block reading a body for /validate. /validate authenticates from headers/cookies only and never reads the body, so reset r.Body to http.NoBody and zero Content-Length at the top of the handler. Fixes the 'Failed to generate token' 500 with the sidecar inline. Refs #1652
- TestHandleValidate_DoesNotBlockOnPhantomBody: reproduces the gateway 500 (a /validate subrequest with a declared Content-Length but a body that never arrives must not hang the fallback proxy). Fails without the http.NoBody fix. - Unit tests for extractBearer precedence + A2A rule, canonicalAuthMethod, serverNameFromOriginalURL, validateSecretKey (weak/strong), recordEnabled (fail-closed), scope snapshot group->scope mapping, and the scope resolver's A/B/C cases incl. the user-generated sentinel. Refs #1652
…oad tests Adds a tiny, fast Go streamable-http MCP server (single 'echo' tool) as an opt-in benchmark upstream so end-to-end gateway load tests are bounded by the /validate auth check, not by a slow upstream (a heavy app endpoint hides the auth-check cost; a fast MCP server exposes it). - servers/pingmcp/: vendored source (canonical repo: https://github.com/aarora79/pingmcp) - docker-compose.yml: pingmcp-server service behind the 'benchmark' profile (does not start by default); reached in-cluster as pingmcp-server:8100. - cli/examples/pingmcp.json: registration config. Registering it also requires adding pingmcp-server to SSRF_ALLOWED_HOSTS. Refs #1652
… and CI Productionizes the go-validate /validate fast-path sidecar across all deployment surfaces. OPT-IN everywhere and non-breaking: when disabled (the default), VALIDATE_UPSTREAM_URL is unset so nginx /validate stays on the Python auth-server exactly as before, and no sidecar container is created. Terraform/ECS (terraform/aws-ecs): - go_validate_enabled / go_validate_image_uri / go_validate_audience / validate_upstream_url vars (root + module, wired via main.tf). - go-validate as a sidecar container in the auth-server task (opt-in), sharing the task namespace (AUTH_FALLBACK_URL=localhost:18888); reuses SECRET_KEY, nginx marker, and DocumentDB creds/secrets; JWKS/issuer derived from Keycloak. - Service Connect alias + SG ingress on 8899; registry VALIDATE_UPSTREAM_URL env. - terraform validate passes. Helm/EKS (charts): - goValidate sidecar container + Service port 8899 on the auth-server chart (guarded by goValidate.enabled), reusing the same secretRefs. - registry VALIDATE_UPSTREAM_URL secret (only when app.validateUpstreamUrl set, so env[] indices don't shift); reserved-env-names updated; stack guidance. - New helm-unittest suites (sidecar on/off, service port, secret emission); full suite 193 tests pass. CI/ECR: - buildspec.yml + build-and-push-all.sh + build-config.yaml + release-images.yml build and push mcp-gateway-go-validate (context go-validate/). Refs #1652
…CP protocol go-validate (18.8% -> 55.6% stmt coverage; remainder is network/DB wiring exercised by the live parity runs): - verifyRS256 all branches (valid, bad sig/expired -> 401, unknown kid/wrong iss/wrong aud -> fallback, non-JWT); audContains string+array. - mintRegistryUIToken / mintMCPProxyToken round-trip (HMAC verified, claim shapes/audiences/token_use asserted); mintInternal empty-subject fail-closed. - parseKey (JWK->rsa.PublicKey); buildMongoURI (disabled + escaped creds). - full fast-path handler (verify->map->mint->headers) + recognized-invalid 401 + health/metrics. pingmcp (62.5%): initialize (+session header, protocol echo), tools/list, tools/call echo, ping, notifications/initialized 202, unknown-method error, GET 405. Refs #1652
…ee reference points Ergonomics + the config-surface checklist for the sidecar params: - Auto-derivation: go-validate now derives JWKS_URL and VALIDATE_ISSUER at runtime from KEYCLOAK_URL / KEYCLOAK_EXTERNAL_URL / KEYCLOAK_REALM (already present in every deployment) when unset. Operators now only set enabled + audience (aud can't be auto-derived; Keycloak varies it). Explicit values win; unit-tested (derive / explicit-wins / no-keycloak-fallback). - docker-compose: pass KEYCLOAK_* to the sidecar so derivation works there too. - Helm auth-server values: document that jwksUrl/issuer auto-derive (override-only). - System Config UI: add validate_upstream_url to registry Settings + CONFIG_GROUPS (Settings -> System Config -> Authentication). - docs/unified-parameter-reference.md: Group 3 rows for VALIDATE_UPSTREAM_URL and the go-validate sidecar params across Docker / Terraform / Helm. Refs #1652
…fastPath) + compose auto-derive - Rename the operator-facing enable switch from the implementation name to the FEATURE name: Terraform go_validate_enabled/image_uri/audience -> validate_fast_path_enabled / validate_fast_path_image_uri / validate_fast_path_audience; Helm goValidate.* -> fastPath.*. Artifact names (go-validate image/dir/container, VALIDATE_* env contract) are unchanged. - Single switch: on ECS, validate_fast_path_enabled=true now BOTH deploys the sidecar AND auto-routes nginx /validate to it (VALIDATE_UPSTREAM_URL derived unless overridden). On Helm, global.fastPath.enabled does the same across the auth-server + registry subcharts (honored via dig for standalone safety). - docker-compose: JWKS_URL now auto-derives from KEYCLOAK_* by default (was a hardcoded fallback); .env.example documents the compose switch (VALIDATE_UPSTREAM_URL: go-validate:8899 = on, auth-server:8888 = off). - Tests: added global-switch cases; full helm suite 180 pass, terraform validate ok. Refs #1652
The unified reference named the three fast-path values but did not tell an operator what to set them to. Add a subsection under Group 3 that: gives the one-liner to decode a real token's aud/iss, states the exact derived formats for issuer and JWKS URL (so they can be verified or overridden), and shows how to confirm the fast path engaged via /metrics (fastpath_ok vs fallback). Refs #1652
…ail-safe Match the Python Keycloak provider exactly and make a broken fast path visible. Verifier parity (was a real gap): - Accept a LIST of issuers (external + internal + localhost realm URLs) and a LIST of audiences, matching on ANY member - Python already accepts all three issuers and [client_id, m2m_client_id, mcp-gateway]. The old single-issuer/single-audience check fell back on browser-login tokens and, worse, accepted 'account'. - Refuse 'account' as an audience even if set explicitly (strip + log): it rides on every realm token, so accepting it is a same-realm cross-client confused-deputy that Python rejects. A looser fast path accepting what Python rejects is a bypass. - Auto-derive both lists from KEYCLOAK_* (issuers from URL/EXTERNAL_URL/REALM, audiences from CLIENT_ID/M2M_CLIENT_ID + mcp-gateway) so operators set nothing. Explicit VALIDATE_ISSUER/VALIDATE_AUDIENCE take comma/space lists to override. Fail safe, loudly (logs + metrics), never crash: - Metrics gauges govalidate_fastpath_ready and govalidate_jwks_healthy plus govalidate_jwks_refresh_failures_total; ready=1 + jwks_healthy=0 = degraded. - ERROR log on the healthy->degraded JWKS transition, WARN on repeat, INFO on recovery; WARN FALLBACK-ONLY at startup naming exactly which vars are unset. - Fallback to Python stays correct throughout (safe), it is just now visible. Wiring: - compose + ECS sidecars now pass KEYCLOAK_CLIENT_ID/M2M so audiences derive; ECS switched from a single hardcoded issuer to full auto-derivation. Helm already passed KEYCLOAK_* via envFrom (no template change). - Examples enable the fast path by default: .env.example (compose) already routed to the sidecar; terraform.tfvars.example now sets validate_fast_path_enabled=true (the variable DEFAULT stays false, so upgrading stacks are unaffected). - Docs: unified reference + SECURITY_GUIDELINES (never accept an IdP universal default audience; a re-implemented verifier must reproduce the same allowlists). - Tests: multi-issuer/multi-audience match, account rejection, missingReason, degraded metrics, JWKS refresh-failure counter. go test 65.9%%, helm 180, tf ok. Refs #1652
ECS uses Cognito, so the fast path now supports it alongside Keycloak, mirroring auth_server/providers/cognito.py exactly. - Provider select: AUTH_PROVIDER (the auth-server already sets it) picks the verifier; auto-detected from COGNITO_USER_POOL_ID / KEYCLOAK_URL when unset. Only keycloak + cognito are fast-pathed; other IdPs stay fallback-only (safe). - Cognito verify (cognito.go): issuer https://cognito-idp.<region>.amazonaws.com/ <pool>, JWKS at /.well-known/jwks.json. Access tokens only (token_use=access; id/login tokens defer to Python). Access tokens are client_id-bound (no aud): client_id must be in the allowlist (web + IDE + M2M ids), with a '*' M2M wildcard honored for machine tokens (no username) only - never widening user tokens. All auto-derived from COGNITO_* / AWS_REGION. - Scopes match server.py: cognito:groups -> group->scope mapping (same DocumentDB path); no-group / machine token -> the token's own scope claim. - Refactor: extract parseVerifyDecode() (shared RS256 crypto half); handleValidate now branches via resolveFastPath() -> resolveKeycloak / resolveCognito, sharing the header/mint tail. Keycloak path behavior unchanged. - Wiring: ECS sidecar + compose now pass AUTH_PROVIDER + AWS_REGION + COGNITO_*; Helm sidecar already gets them via envFrom the auth secret (no template change). Enablement is the SAME switch (validate_fast_path_enabled / fastPath) - the provider is auto-detected, no new enable param needed. - Loud fail-safe + metrics are provider-aware (missingReason, startup log). - Tests: verifyCognito (access/id/wrong-iss/bad-sig/client-id allowlist/M2M wildcard), resolveCognito scope sources, cognito config derivation + wildcard. go test 69.2%, helm 180, terraform validate ok. Refs #1652
The sidecar now supports Cognito (auto-detected from AUTH_PROVIDER=cognito): derives issuer/JWKS/client-id allowlist from COGNITO_*/AWS_REGION, fast-paths access tokens, scopes from cognito:groups or the token scope claim. Same enable switch; VALIDATE_AUDIENCE does not apply to Cognito. Refs #1652
The registry task definition mounts aws_secretsmanager_secret.embeddings_idp_client_secret (EMBEDDINGS_AUTH_MODE=idp), but that secret's ARN was missing from the ecs_secrets_access execution-role policy. Result: ResourceInitializationError (AccessDenied on secretsmanager:GetSecretValue) at task init, so registry tasks could never start (crash loop, 0/2 running) whenever the embeddings-idp secret is created. Add the ARN to the policy (unconditional, alongside embeddings_api_key). Unrelated to the go-validate fast path; surfaced during the same terraform apply.
…'s TLS On ECS (DOCUMENTDB_USE_TLS=true) the sidecar reached DocumentDB but the TLS handshake failed: 'x509: certificate signed by unknown authority'. DocumentDB serves a cert signed by the Amazon RDS CA, which is not a public root, so the Mongo driver never trusted it -> the scope snapshot never loaded -> every group-bearing (Cognito user) token fell back to Python instead of fast-pathing. - scopes.go: when tls=true, append &tlsCAFile=<DOCUMENTDB_TLS_CA_FILE> (default /app/certs/global-bundle.pem), mirroring the Python containers exactly. - Dockerfile: bake the Amazon DocumentDB CA bundle into the image at that path (vendored global-bundle.pem, same file the Terraform stack ships). - test: buildMongoURI now asserts tls=true carries tlsCAFile, and honors a custom DOCUMENTDB_TLS_CA_FILE (URL-encoded). Verified: go test 69.5%, docker build resolves the CA COPY. Fixes the 'scope resolver: refresh failed ... unknown authority' loop seen in the go-validate ECS logs. Refs #1652
*.pem is gitignored (secret-guard), so the vendored bundle from the prior commit would be missing on a fresh checkout and break the image build. Download the Amazon RDS/DocumentDB global-bundle.pem in the build stage (same public source the Python entrypoints use) and COPY it into the final image at /app/certs/global-bundle.pem. Verified with a local docker build. Refs #1652
Switch the build stage from Docker Hub golang:1.24-alpine to public.ecr.aws/docker/library/golang:1.24-alpine, matching the repo convention (docker/Dockerfile.metrics-db) and avoiding Docker Hub rate limits. Build stage only; the runtime image remains distroless/static. Verified with docker build. Refs #1652
…ted) After the TLS/CA fix the handshake succeeded but auth failed: 'unable to authenticate using mechanism SCRAM-SHA-256: Unsupported mechanism'. Amazon DocumentDB v5.0 only supports SCRAM-SHA-1; other MongoDB-compatible backends support SCRAM-SHA-256. Mirror registry/utils/mongodb_connection.py: select the mechanism from STORAGE_BACKEND (documentdb -> SCRAM-SHA-1, else SCRAM-SHA-256), and pass STORAGE_BACKEND to the sidecar in ECS + compose (it was missing). Test covers both mechanisms. go test green, terraform validate ok. Refs #1652
go-validate is the /validate fast-path sidecar that runs inside the auth task, so deploying it means build+push its image (make build-push IMAGE=go_validate, already wired via build-config.yaml) then force a new auth deployment to pull it. - --service auth and --service both now also build the go-validate image before the auth force-new-deployment (the sidecar ships in the auth task). - Adds --service go-validate (aliases: go_validate/govalidate) to rebuild only the sidecar and redeploy auth. - Dynamic step counting; usage/help updated. bash -n clean. Refs #1652
… unset
The prior SCRAM fix keyed only on STORAGE_BACKEND, which requires a terraform
apply to reach the sidecar. On the live ECS task (rev 157) STORAGE_BACKEND was
never wired, so the code defaulted to mongodb-ce -> SCRAM-SHA-256 -> DocumentDB
kept rejecting it ('Unsupported mechanism'), even though the TLS/CA fix worked.
Make it robust: an explicit STORAGE_BACKEND still wins, but when it is unset,
fall back to the TLS signal (DocumentDB always runs with TLS; local mongo-ce does
not) -> tls=true => SCRAM-SHA-1. The sidecar already receives DOCUMENTDB_USE_TLS,
so a plain image redeploy (deploy.sh --service go-validate) now fixes DocumentDB
auth without needing another terraform apply. Test covers heuristic + explicit-wins.
Refs #1652
Extend the fast path to Microsoft Entra ID and Okta, same pattern as Cognito, mirroring auth_server/providers/entra.py and okta.py. Shared machinery reused: RS256 verify (issuer-list + audience-list), group->scope + idp_m2m_clients enrichment, token minting, loud metrics, DocumentDB TLS/SCRAM. Entra (entra.go): - Dual issuers (v2 <login>/<tenant>/v2.0 + v1 sts.windows.net/<tenant>/), JWKS at <login>/<tenant>/discovery/v2.0/keys, all derived from ENTRA_* env. - Accepted audiences: client id + api://<client-id> + ENTRA_APPLICATION_ID_URI. - id_token replay guard: reject on id_token-only claims (nonce/at_hash/c_hash), deferring to Python (never accept a token Python rejects). - Groups from 'groups', or 'roles' for M2M tokens. Okta (okta.go): - Org vs custom-auth-server issuer/JWKS from OKTA_DOMAIN (+ OKTA_AUTH_SERVER_ID). - Accepted audiences: OKTA_CLIENT_ID + OKTA_M2M_CLIENT_ID + OKTA_M2M_ALLOWED_AUDIENCES. - username from 'sub', client id from 'cid', scopes from 'scp'/'scope'. Shared resolveScopes() finalizes scopes exactly like server.py: groups -> group->scope map; else M2M enrichment; else the token's own scope claim. Wiring: sidecar now receives ENTRA_*/OKTA_* in ECS + compose (already gets AUTH_PROVIDER). Provider auto-detected; no new enable param. Tests: verify (valid/v1-issuer/api-aud/id_token-reject/aud-mismatch), roles fallback, claim mapping, scp parsing, config derivation (custom + org auth server), resolveScopes. go test 71.4%, terraform validate ok. Auth0 + PingFederate still deferred (fallback-only) per plan. Refs #1652
Rebaselined on main (merge) surfaced three failing checks, all from this branch:
- Helm render nil-pointer: the stack values set auth-server.fastPath to a
comment-only block (YAML null), which nilled the subchart's fastPath map ->
'nil pointer evaluating interface {}.enabled' at service.yaml. Set it to an
empty map ({}) so subchart defaults survive, and make the templates nil-safe
with (.Values.fastPath).enabled in service.yaml + deployment.yaml. Fixes both
'Helm Unit Tests' and 'Reserved Env Name List Sync' (the latter only failed
because the render crashed; VALIDATE_UPSTREAM_URL is already in the reserved list).
- detect-secrets: regenerated .secrets.baseline via the documented command
(detect-secrets v1.5.0, --slim, same excludes) so the test-fixture 'secrets' in
go-validate/ and charts/*/tests/ are baselined. No .env/real secrets included.
Verified locally: helm unittest 195 pass, reserved-env-sync check passes
(registry 119 / auth 59 / mcpgw 17 all reserved), detect-secrets-hook passes,
go test + terraform validate ok.
A self-contained guide a tester can follow to confirm the /validate fast path works end to end, independently: - confirm the sidecar is healthy (mode=fast-path, jwks_healthy, /metrics) - run the pre-release e2e suite (non-breaking check) - register the bundled pingmcp fast upstream, incl. adding pingmcp-server to SSRF_ALLOWED_HOSTS (required or registration/health-check is blocked) - mint a REAL IdP RS256 token (Keycloak/Cognito) and prove fastpath_ok increments (the .token HS256 falls back by design) - load-test RPS against /pingmcp/ and read fastpath_ok vs fallback - notes on the direct-/validate vs through-gateway measurement difference Refs #1652
Reword the title and intro so the guide reads as general ('anyone') rather than
scoped to an independent tester.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a stdlib-only Go fast-path sidecar (
go-validate/) that fronts the auth-server'sGET /validateauth_requestendpoint. nginx calls/validateon every proxied request; the Python handler runs on a single uvicorn worker and is the throughput ceiling of the authenticated data path (repo baseline: ~172.7 rps @ c=50,tests/stress/results/validate/baseline.json).go-validateserves the steady-state configured-IdP RS256 bearer path in Go and reverse-proxies everything else to the unchanged Python auth-server. It is byte-identical where it answers and zero-risk where it does not — the cutover is a single nginxproxy_passline.Closes #1652.
What it does
X-Internal-Token-Registry-> write the identity headers nginx consumes.kid, or an unconfigured fast path are reverse-proxied toauth-server:8888.GET /validate,GET /health(readiness; degraded when JWKS unhealthy),GET /metrics(plaintext counters).Security invariants baked in
SECRET_KEYfor missing AND weak at startup — fails closed.iss/audfrom config, never from the token.401; unrecognized -> fallback (never a silent allow).Scope (hackathon)
In: RS256 fast path + Python fallback, Docker/compose surface.
Deferred (stays on Python via fallback): federation/admin static tokens, session cookies, OBO exchange, per-tool ACL, rate limiting, audit. Terraform/ECS + Helm/EKS wiring, unified parameter reference, and System Config page are deferred to productionization.
Testing
go build ./...andgo vet ./...pass; static binary ~9.4 MB..scratchpad/(not committed).uvicorn --workers N).Build with Claude Code
Engineered with Claude Code + Claude Opus via the
new-feature-designskill: iterative issue -> design (LLD + diagrams) -> testing plan -> adversarial multi-persona review -> implementation, refined with human feedback.