feat(gateway-proxy): generic-proxy extensions — streaming, static upstream auth headers, caller passthrough - #1714
Open
omrishiv wants to merge 14 commits into
Conversation
… script
Round out the security coverage with tests that compose the REAL layers end to
end (the per-slice suites each cover one layer in isolation; cross-layer bugs
hide at the boundaries).
In-process composite suite (tests/auth_server/unit/test_gateway_proxy_composite.py):
- mint -> verify -> handler trust chain: a /validate-minted generic token
verifies on the handler and confines the outbound to the bound
(entity_type, registered_path); cross-type and sibling replays are rejected.
- mint discriminator composed: driving the real _attach_mcp/_attach_generic pair
on one response proves exactly one token per request kind (no double-mint).
- render <-> authz coherence: the nginx block a render emits for an entity uses
the SAME /{entity_type}/{path} shape the token binds and the handler confines,
so an authored scope, the minted token, and the rendered route all agree.
- feature-latch composed: a valid token still 503s when the egress self-check
latch is off (runtime gate composes with the token gate).
E2E smoke script (tests/integration/test_gateway_proxy_e2e.sh): real HTTP against
a running stack. Registration-time SSRF reject (metadata IP + loopback) runs
regardless of the feature flag; live-routing checks (response-header allowlist
drops backend Set-Cookie, marker-spoof on /api/ does not route to an attacker
upstream) run only when GATEWAY_GENERIC_PROXY_ENABLED=true.
Add opt-in response streaming to the generic-proxy hop so a long-lived SSE / chunked upstream (e.g. an LLM proxied as a custom entity type) flows to the client incrementally instead of being buffered. - ProxyableMixin gains proxy_streaming (federation-stripped like the other proxy fields); projected in the agent/skill/server/custom list_proxied feeds and threaded through the nginx render. - Streaming routes render proxy_buffering off + a long proxy_read_timeout (GATEWAY_GENERIC_STREAM_READ_TIMEOUT_SECONDS) and set the $generic_streaming marker; buffered routes are unchanged. - The marker is forwarded on the shared /validate subrequest and bound into the signed generic-proxy token, so the hop streams only on the verified claim -- never a forgeable inbound header. - generic_proxy returns a StreamingResponse driven by aiter_bytes() when the claim is set, holding the httpx client/stream and the concurrency semaphore alive via a BackgroundTask cleanup; read timeout disabled (connect timeout kept), no response-size cap while streaming. - Config knob wired across compose, Helm (values + deployment + reserved-names + extra_env index test) and Terraform (root + module + ecs env). Tests: token mint/verify carries the streaming claim; nginx render emits buffering-off only for streaming entities. 251 generic-proxy/mixin unit tests + 180 helm unittests pass; mypy/bandit/ruff clean.
Let a proxied non-MCP entity (skill / custom entity, and MCP servers / agents that inherit ProxyableMixin) present a static credential to its backend -- e.g. an API key for an LLM proxied as a custom type -- without the secret ever touching the nginx config or logs. Data path (mirrors the egress credential vault's trust boundary): - ProxyableMixin gains custom_headers_encrypted / custom_header_names / custom_headers_updated_at; CustomHeaderEncrypted moved here (re-exported from core.schemas). All are federation-stripped via PROXY_FIELD_NAMES (plus the plaintext custom_headers key as defense-in-depth). - Render emits a bool $generic_has_upstream_auth marker (never the secret); /validate forwards X-Generic-Has-Upstream-Auth and binds a has_upstream_auth claim into the signed generic-proxy token, so the hop acts only on the verified claim, never a forgeable header. - New registry POST /internal/generic-upstream-headers (sibling of the egress-token vend): validate_internal_auth + re-verified X-Internal-Token- Generic, entity-type-agnostic resolution, upstream cross-check, decrypts and returns the headers. New registry-side verify_generic_proxy_token. - generic_proxy fetches + injects on the signed claim and FAILS CLOSED (502, no unauthenticated forward) if the vend fails -- buffered and streaming paths both. - Registration: skill / custom-entity create accept a plaintext custom_headers list, validated (reserved-name block + count cap via the shared validate_custom_headers) and encrypted; write-only on reads. Security hardening: - Credential-misdirection guard: clear_upstream_headers_on_repoint clears stored headers on all four update paths (skill/custom/server/agent) when the effective target host changes, so the old host's secret is never sent to a new host. Uses effective_proxy_target (routability-agnostic) so it fires even for a disabled / not-yet-enabled server. - Skill PUT strips header fields (no plaintext-secret $set; header rotation deferred to a dedicated endpoint, matching the MCP-server pattern). Tests: shared header-policy validation, vend endpoint (resolve + upstream cross-check + fail-closed), token claim bind, nginx marker render, federation strip, and the repoint/disabled-server regression. mypy, bandit, ruff clean.
…able flag
Extend the generic-proxy upstream-header feature so a caller can supply or
override upstream auth headers, opt-in per registered header:
- Each registered header is {name, value?, overridable?}. overridable=false+value
= FIXED operator credential (immutable); overridable=true+value = operator
DEFAULT the caller may override; overridable=true+no value = caller-only slot.
Authorization is allowed ONLY as overridable (a fixed operator bearer belongs in
the egress vault).
- Registered overridable names form the caller allowlist; the hop forwards a
caller-supplied header only if its name is registered-overridable, with the
caller's value winning over an operator default.
Backend: validate_custom_headers + encrypt_custom_headers_in_server_dict handle
the overridable field; new custom_header_overridable_names stored field (in
PROXY_FIELD_NAMES + list_proxied projections + repoint-clear); vend endpoint
returns the allowlist; the auth_server hop merges operator defaults + caller
passthrough and enforces the A2A equal-token guard on Authorization.
Security hardening:
- RESERVED_CUSTOM_HEADER_NAMES now covers the full gateway-internal header set
(X-Internal-Token*, X-User/X-Scopes/X-Groups/..., X-Generic-* markers,
X-Resolved-*, X-Upstream-Url, X-Body*) so none can be registered as a custom
header and exfiltrated to a registrant-controlled backend.
- The generic hop strips the internal identity/token set unconditionally on egress
(_strip_generic_internal_headers), independent of the denylist.
- The vend backstops BOTH operator defaults and the overridable allowlist against
the reserved set (Authorization carve-out).
…s + custom entities
The create path was previously the only way to set a proxied entity's upstream
custom headers (the general PUT deliberately strips header fields to avoid a
plaintext-secret $set). Add a dedicated, narrowly-scoped rotation surface so
headers can be updated after registration — the mirror of PATCH
/servers/{path}/auth-credential.
- Shared build_custom_headers_storage_fields(): validate + encrypt a plaintext
header list into the four storage fields (custom_headers_encrypted,
custom_header_names, custom_header_overridable_names, custom_headers_updated_at).
An empty/None list CLEARS all headers (flips the has_upstream_auth render marker
off and makes the vend return empty).
- PATCH /api/skills/{path}/upstream-headers: owner-or-admin + modify_skill scope
(same dual gate as PUT), CSRF-protected.
- PATCH /api/custom/{type}/{uuid}/upstream-headers + update_record_upstream_headers:
type-level modify scope + per-record owner-or-admin, CSRF-protected.
Both endpoints re-run the full header policy (reserved/internal-name deny, count
cap, fixed-Authorization reject), never persist or echo plaintext, and set headers
for the CURRENT target (a concurrent repoint stays consistent — the vend reads
target + headers atomically). Also strip custom_header_overridable_names from the
skill PUT payload.
…tities Add a shared UpstreamHeadersField (formFields) implementing the per-header overridable model (fixed operator credential / operator default the caller may override / caller-only passthrough slot), wired into the skill and custom-entity register/edit forms (shown only when the entity is proxied). - Create sends custom_headers inline on the create payload; edit does the general PUT then the dedicated rotation PATCH (skill: Dashboard.performSkillSave; custom: CustomEntityTab via useCustomEntities.rotateRecordHeaders), gated on is_proxied. Empty header sets are omitted from create. - Write-only value convention (mirrors the 3LO egress client_secret and the MCP-server custom-header edit): on edit, rows load with BLANK values and a blank value PRESERVES the stored ciphertext by name. build_custom_headers_storage_fields gains an existing_encrypted param for the preserve-by-name merge; the rotation endpoints pass the entity's current custom_headers_encrypted. - Client-side policy hint (upstreamHeaderRowError) mirrors the backend reserved-name deny (incl. all gateway-internal headers), the fixed-Authorization reject, and the value-less-non-overridable reject — a fail-fast UX aid, not the security boundary (the backend re-validates).
…n endpoint
The comment described header rotation as a future follow-up; the dedicated
PATCH /skills/{path}/upstream-headers endpoint now exists, so reference it.
Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>
Security / defense-in-depth:
- Wire _strip_generic_internal_headers as an egress backstop on the caller
baseline (before upstream-auth injection) so the documented backstop is on the
real path, independent of the positive allowlist; update the handler test to
assert the true allowlist+strip pipeline it now runs.
- Add x-entity-path and x-original-method to RESERVED_CUSTOM_HEADER_NAMES so
registration and the vend backstop align with the egress strip set.
- Couple the generic-upstream-headers vend timeout to the registry transient-retry
budget (was a hardcoded 10s) and log a WARNING on the fallback branch.
Streaming reliability / observability:
- Bound time-to-first-byte and inter-chunk idle by the stream read-timeout knob at
the auth-server hop (previously applied only to nginx), so a connect-then-stall
upstream cannot hold a stream slot for the full absolute duration.
- Emit metrics: generic_proxy_slot_rejected_total{pool} for capacity 503s and
generic_proxy_stream_outcome_total{outcome} (started / completed /
duration_timeout / byte_cap / upstream_error).
- Add runtime tests for _generic_proxy_streaming (chunks, byte-cap, setup error,
capacity rejection) asserting slot release and outcome metrics on every path.
Config / infra:
- Surface the five stream settings in CONFIG_GROUPS (System Config UI).
- Gate the registry :8091 internal vend listener (Service + NetworkPolicy) on the
generic-proxy feature OR egressAuth, so static upstream headers work on
Kubernetes without the per-user egress vault.
- Normalize the module-level registry_extra_env reserved-name check with
upper()/trimspace() to match auth_server_extra_env and the root module.
Frontend:
- Block submit when an upstream-header row is invalid (reserved name / fixed
Authorization / value-less non-overridable), edit-mode aware so a blank
write-only value still preserves the stored ciphertext; unique per-row
aria-labels; trim non-blank header values.
Docs:
- Record streaming, encrypted static upstream auth headers, and the caller
passthrough overridable slot (the second ingress-header exception) with their
guards in gateway-generic-proxy.md and egress-auth-design.md.
…and mid-stream error The streaming hop released its concurrency slot on client disconnect and mid-stream upstream failure but recorded no terminal outcome metric, so the started counter drifted above the sum of terminals and in-flight could not be derived from counter deltas. - Record a client_closed outcome when the request is cancelled before response headers (setup CancelledError) and when the body generator is closed mid-stream (GeneratorExit / CancelledError at the yield); the slot is still released by the idempotent cleanup. - Record upstream_error when the upstream drops the connection after headers were already sent (mid-body httpx error), which previously fell through untracked. - Add client_closed to the outcome label set. Tests: add streaming idle/duration-timeout, mid-stream client-disconnect, and mid-stream upstream-error cases, each asserting slot release plus the recorded outcome -- completing terminal-path coverage (completed / duration_timeout / byte_cap / upstream_error / client_closed).
omrishiv
changed the title
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>
The _proxy_allowlist / _skill_allowlist / _builtin_airegistry_tools_allowlist functions are @lru_cache(maxsize=1) over gateway_proxy_allow_private_targets and the SSRF allow/deny lists, cached process-wide. A test that enables allow_private and populates the cache (e.g. test_proxy_mixin's test_allowed_when_flag_set) leaks that relaxed policy to later tests on the same xdist worker, so the server-registration SSRF-rejection tests intermittently see private/loopback/CGNAT targets as allowed and fail with 'DID NOT RAISE UrlValidationError'. Add an autouse fixture that clears the caches around every test, mirroring _reset_os_environ.
Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(gateway-proxy): generic-proxy extensions — streaming, static upstream auth headers, caller passthrough
Closes: #1565
Summary
Extends the already-merged generic reverse proxy (which serves non-MCP registry entities — skills, agents, custom types — through the gateway) with four capabilities plus security hardening. The feature still ships disabled by default (
GATEWAY_GENERIC_PROXY_ENABLED=false) and fails closed; nothing changes for existing deployments until an operator enables it and deploys the egress policy.What's in this PR
Per-entity response streaming (SSE / chunked) — a proxied entity with
proxy_streaming=truestreams its upstream response to the client incrementally instead of buffering (e.g. an LLM proxied as a custom entity). Isolated concurrency pool, bounded acquire wait, absolute-duration ceiling, idle read-timeout (time-to-first-byte + inter-chunk), and a raw-byte cap;proxy_buffering offon the nginx route. Streaming is driven only by a signed token claim bound at/validatefrom a server-set nginx marker — never a forgeable inbound header.Static upstream auth headers — an operator credential (e.g. a backend API key) is stored encrypted at rest (
custom_headers_encrypted, SECRET_KEY-derived Fernet), vended registry→auth-server over a service-token-gated internal endpoint, and injected on egress. The secret never enters nginx config or logs and is stripped from list/card responses. Fail-closed: if the vend fails, the request is refused (502), never forwarded unauthenticated. A vend-time canonical-URL cross-check and a repoint credential-clear prevent misdirection to a different backend.Caller header passthrough (per-header
overridableflag) — each registered header is fixed (operator credential, immutable) / operator-default (caller may override) / caller-only slot. Only registered-overridable names may be forwarded from the caller;Authorizationis allowed only as overridable and is guarded by an A2A equal-token check so the gateway's own credential can never leak to the backend.Rotation endpoints + editor UI — dedicated CSRF-protected
PATCH .../upstream-headersendpoints (skill + custom entity) to rotate headers after registration, and a sharedUpstreamHeadersFieldeditor wired into the create/edit forms (shown only when the entity is proxied) with a write-only value convention (blank = keep stored ciphertext).Security hardening — an unconditional egress backstop that strips the full gateway-internal header set (
_strip_generic_internal_headers),RESERVED_CUSTOM_HEADER_NAMEScovering that set so it can't be registered, and terraform/helm reserved-name guards.Design notes / invariants
guarded_async_client.docs/unified-parameter-reference.mdandCONFIG_GROUPSupdated.New configuration
Six settings, wired across
.env.example, all compose files, terraform (root + module + task def), helm (values + deployment + reserved-env), and documented indocs/unified-parameter-reference.md:GATEWAY_GENERIC_STREAM_MAX_CONCURRENCY8GATEWAY_GENERIC_ACQUIRE_TIMEOUT_SECONDS5GATEWAY_GENERIC_STREAM_MAX_DURATION_SECONDS3600GATEWAY_GENERIC_STREAM_MAX_BYTES104857600GATEWAY_GENERIC_STREAM_READ_TIMEOUT_SECONDS3600proxy_read_timeoutNew per-entity fields (federation-stripped):
proxy_streaming,custom_headers_encrypted,custom_header_names,custom_header_overridable_names,custom_headers_updated_at.Kubernetes note: the registry
:8091internal vend listener (Service + NetworkPolicy) is gated on the generic-proxy feature OR the per-user egress vault, so static upstream headers work without enablingegressAuth.Observability
Two OTel counters on the auth-server hop:
mcpgw_registry_generic_proxy_slot_rejected_total{pool}— capacity 503s.mcpgw_registry_generic_proxy_stream_outcome_total{outcome}—started | completed | duration_timeout | byte_cap | upstream_error | client_closed(all terminal paths, so in-flight = started − Σterminals).Testing
test_gateway_proxy_composite.py), streaming-handler runtime tests (test_generic_proxy_streaming.py— chunks, byte-cap, setup/mid-stream error, capacity, idle-timeout, client-disconnect, each asserting slot release + outcome metric), upstream-header vend + rotation route tests, and an E2E smoke script (test_gateway_proxy_e2e.sh, self-skips unless the feature is deployed; reads its bearer from--token-file).fmtclean; frontendtscclean + 35 jest tests.ruff/banditare the CI gate (not in the local venv).By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.