Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -606,6 +606,11 @@ Every event on `/v1/sessions/{id}` and `/v1/events` carries:
426 sends it straight to HTTPS. Nothing is forwarded, the Finisher sees an allow with
status 426, and `/v1/usage` does not count it as an error. On by default;
`listener.decline_responses_websockets: false` forwards the upgrade instead.
**A 403 with `error.kind: "destination_private"` is the egress guard's
answer, not the upstream's** (`listener.egress_guard`, off by default): the
destination resolved into loopback, link-local, private or this host's own
address space and no permit covered it. The Finisher sees an allow with
status 403, like the 426, and `/v1/usage` does not count it as an error.
**A request whose body could not be buffered is the other exception:** it never
reached the pipeline, so its request row carries no
parse and no Finisher runs. Over the forward proxy's `maxRequestBodySize` (32 MiB)
Expand Down Expand Up @@ -636,7 +641,8 @@ Every event on `/v1/sessions/{id}` and `/v1/events` carries:
- `requestedHost` — present only when a plugin redirected the request (`pctx.Redirect`): the host the client asked for. `host` is where the request actually went, and usage, the cost ledger and pricing all follow `host`. agentop's detail pane shows both on a `redirected:` line.
- `inference.requestedModel` — present only when a plugin changed the model (`pctx.SetRequestModel`): the model the client asked for. `inference.model` is the model the request was sent for, which settlement prices; agentop's detail pane shows both on a `model:` line.
- `bytesUp`, `bytesDown` — **body** bytes, one direction per row: a request row carries `bytesUp`, the body **as forwarded** (so a `tool-prune`-style rewrite is reflected, since the count is taken after the pipeline ran), and a response row carries `bytesDown`, the body **as it came back from the destination** — not as delivered, because extproc records the row before it emits a response mutator's replacement. Headers are not counted. A tunnel close is the one row carrying both. **Zero means the listener counted nothing, not that the body was empty**, and there is no third value for "unknown": every listener buffers bodies only when a plugin asks, so an unbuffered request reports zero. A response relayed straight through is counted, not zero — both proxies record the row after the copy (#1251's fix), so the bytes they relayed through their own process are on it. The one arm that still reports ↓0 is extproc with a body-free pipeline: it asked Envoy for body mode NONE, so the bytes it never saw are bytes it asked not to see — `core/listener/parity`'s `TestDivergence_InboundResponseBodyCountWhenNoPluginReadsIt` pins that disagreement with its reason. agentop renders a zero side blank rather than as `0B`, and that is why its BYTES column can be trusted as a size when it shows one. Everywhere else the three listeners count the same thing for the same response, which took some doing: extproc sums the chunks Envoy hands it, while the proxies count off the upstream body at one choke point each, so every arm — buffered, block-by-block SSE, or chunk copy — reports the same figure. `core/listener/internal/bodycount` and `core/listener/parity`'s bytes fixtures hold that line.
- `tunnel`, `tunnelReason` — an opaque CONNECT (or transparent-redirect) tunnel records two rows sharing a `requestId`: the open (`phase: "request"`, `tunnelReason` saying why the bytes stayed opaque) and, when the tunnel ends, the close (`phase: "response"`). The close carries the CONNECT's own `statusCode` — 200, or 502 with the dial error in `error` when the destination could not be reached (`tunnelReason: "dial-failed"`) — plus `durationMs` for how long the tunnel stayed open and, in `bytesUp`/`bytesDown`, the bytes it carried each way (up = client to destination; opaque bytes have no body/header split, so these are everything that crossed). It is not the destination's status: that travels inside the client's end-to-end TLS. A bridged tunnel's open is recorded with its first decrypted request — in that request's session, directly before it, stamped with its time — and records no close, because the request carries its own response; a bridged tunnel that recorded no request gets its open and a close when it ends. Tunnel rows are kept out of `/v1/usage`: a tunnel's lifetime is not a request latency.
- `tunnel`, `tunnelReason` — an opaque CONNECT (or transparent-redirect) tunnel records two rows sharing a `requestId`: the open (`phase: "request"`, `tunnelReason` saying why the bytes stayed opaque) and, when the tunnel ends, the close (`phase: "response"`). The close carries the CONNECT's own `statusCode` — 200, or 502 with the dial error in `error` when the destination could not be reached (`tunnelReason: "dial-failed"`) — plus `durationMs` for how long the tunnel stayed open and, in `bytesUp`/`bytesDown`, the bytes it carried each way (up = client to destination; opaque bytes have no body/header split, so these are everything that crossed). It is not the destination's status: that travels inside the client's end-to-end TLS. A bridged tunnel's open is recorded with its first decrypted request — in that request's session, directly before it, stamped with its time — and records no close, because the request carries its own response; a bridged tunnel that recorded no request gets its open and a close when it ends. Tunnel rows are kept out of `/v1/usage`: a tunnel's lifetime is not a request latency. `tunnelReason: "dest-private"` with a 403 close and `error.kind: "destination_private"` is the egress guard refusing the destination before any dial — never confused with `dial-failed`, which attempted.
- `dialAddr` — the address the egress guard checked and the proxy dialed, `"ip:port"`, on a tunnel's open row when `listener.egress_guard` is on. With `host` it is the audit trail against DNS rebinding: under the guard the proxy resolves once, checks that answer, and dials exactly it, so the two fields together say "asked for the name, reached this address". Absent when the guard is off and on rows that dialed nothing.
- `httpMethod`, `httpPath` — the HTTP verb and path, so a request no parser recognized is still identifiable rather than showing only a host. Distinct from the `method` inside `a2a` / `mcp`, which is a protocol method name. On an opaque tunnel `httpMethod` is `CONNECT` and `httpPath` is absent — opaque bytes carry no request line. The path is query-stripped and percent-decoded, so query-borne credentials never reach the timeline, but a secret in a path *segment* (a bot token, a webhook path) does survive on this unauthenticated surface — worth knowing before exporting events off-box.
- `subagent` — present on rows from a Claude Code subagent or agent-team teammate: the `X-Claude-Code-Agent-Id` the request carried. For a subagent that is also the `<id>` in `~/.claude/projects/<dir>/<session>/subagents/agent-<id>.meta.json`; for a teammate the join is unverified. Resolved by `Context.Subagent`, pinned with the User-Agent by `ResolveClient` so no plugin can re-file a row, and **refused rather than sanitised** when unusable (control runes, over 128 bytes), because it is a join key and a cleaned-up id joins nothing. It does not move the row: a subagent's requests stay in their parent's session, so the header must not be added to `session.id_headers`. **It names an agent, it does not classify the row:** `inference.agentRole` stays the field that says whether a row is a subagent's, and a teammate's row is `agentRole: main` with `subagent` set — Claude Code writes `cc_is_subagent=true` only for `agentType: subagent`.
- `parentSubagent` — the agent that started `subagent`, from `X-Claude-Code-Parent-Agent-Id`, resolved with it in one memo (`Context.ParentSubagent`) and under the same rules. Claude Code sends it only when the spawner was itself an agent, so **absent on a row that names a `subagent` means the conversation started it**, not "unknown". The two keys are the tree's node and its edge.
Expand Down
39 changes: 37 additions & 2 deletions cmd/cortex/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ import (

// Only HTTP listeners are compiled in: no extproc
// (no gRPC, no envoy types).
"github.com/rossoctl/cortex/core/listener/egressguard"
"github.com/rossoctl/cortex/core/listener/forwardproxy"
"github.com/rossoctl/cortex/core/listener/reverseproxy"
"github.com/rossoctl/cortex/core/listener/skiphost"
Expand Down Expand Up @@ -799,6 +800,28 @@ func main() {
slog.Info("mTLS disabled (no mtls block in config)")
}

// Egress guard (#1484): when listener.egress_guard arms it, every outbound
// dial the forward proxy makes — CONNECT tunnels, plain and bridged
// requests, transparent redirects, skip_hosts traffic included, because a
// bypass must not become an SSRF bypass — resolves once, refuses loopback,
// link-local, private and host-local destinations with 403
// destination_private, and dials the checked address. Built before the TLS
// bridge below so the re-origination client's dialer is guarded too: the
// bridge must not become the path around the guard. Off unless configured
// (a sidecar's ordinary upstreams live in exactly the refused space).
var egress *egressguard.Guard
if cfg.Listener.EgressGuard != nil && cfg.Listener.EgressGuard.Enabled {
g, gerr := egressguard.New(cfg.Listener.EgressGuard.PermitCIDRs)
if gerr != nil {
fatalf("listener.egress_guard: %v", gerr)
}
egress = g
// The "on" announcement is logged down in the forward-role block, not
// here: the guard is only ever enforced through the forward proxy, and
// a reverse-only deployment must not boot announcing an enforcement no
// listener applies.
}

// TLS bridge: when enabled, the forward proxy terminates agent outbound
// TLS so the outbound pipeline sees decrypted HTTPS. Constructed
// here and set on fpSrv below (mirroring fpSrv.SkipHosts / fpSrv.Shared).
Expand Down Expand Up @@ -886,7 +909,15 @@ func main() {
fatalf("tls-bridge upstream_ca_bundle read failed: %v", err)
}
}
up, uerr := tlsbridge.NewUpstreamClient(extra, cfg.TLSBridge.UpstreamInsecure)
// The bridge's re-origination client dials through the egress guard
// when one is armed, so a bridged request is checked exactly like a
// plaintext one. No option when the guard is off: nil DialContext is
// the transport's default dialer.
var upOpts []tlsbridge.UpstreamOption
if egress != nil {
upOpts = append(upOpts, tlsbridge.WithDialContext(egress.DialContext))
}
up, uerr := tlsbridge.NewUpstreamClient(extra, cfg.TLSBridge.UpstreamInsecure, upOpts...)
if uerr != nil {
fatalf("tls-bridge upstream client failed: %v", uerr)
}
Expand Down Expand Up @@ -966,10 +997,14 @@ func main() {
// declared here so shutdown can close it whether or not the forward role ran.
var transparentLn *net.TCPListener
if roles[config.RoleForward] {
fpSrv, ferr := forwardproxy.NewServer(outboundH, sessions, fpMTLS)
fpSrv, ferr := forwardproxy.NewServer(outboundH, sessions, fpMTLS, forwardproxy.WithEgressGuard(egress))
if ferr != nil {
fatalf("creating forward proxy: %v", ferr)
}
if egress != nil {
slog.Info("egress guard on: refusing loopback, link-local, private and host-local destinations",
"permit_cidrs", cfg.Listener.EgressGuard.PermitCIDRs)
}
// SkipHosts: outbound destinations that bypass the pipeline AND
// session recording entirely. See ListenerConfig.SkipHosts for the
// motivating case (chatty observability sidecars evicting the
Expand Down
43 changes: 43 additions & 0 deletions core/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -944,6 +944,49 @@ type ListenerConfig struct {
// A pointer for the reason SessionConfig.Enabled is one: with a plain bool,
// "unset" and "false" are the same value, and the default could not be on.
DeclineResponsesWebSockets *bool `yaml:"decline_responses_websockets,omitempty" json:"decline_responses_websockets,omitempty"`

// EgressGuard arms the SSRF dial guard on the forward proxy's every
// outbound dial (#1484): each destination is resolved once, refused with
// 403 destination_private when it lands in loopback, link-local (cloud
// metadata, 169.254.169.254), RFC 1918 / unique-local space, RFC 6598
// CGNAT (100.64.0.0/10), or one of this host's own interface addresses —
// the addresses themselves, not their on-link subnets — and dialed at the
// checked address, so DNS rebinding cannot re-answer between check and
// connect. CONNECT tunnels, plain and bridged requests, transparent
// redirects and skip_hosts traffic are all covered: a bypass must not
// become an SSRF bypass.
//
// OFF BY DEFAULT, deliberately: a sidecar's ordinary upstreams — Keycloak,
// in-cluster services, a laptop's local model — live in exactly the
// refused space, so defaulting this on would break every existing
// deployment. An operator who wants the hard boundary (or runs the MOCA
// shape, where the sandbox's egress is meant to be public-only) enables it
// and lists the private prefixes still allowed in permit_cidrs.
//
// permit_cidrs covers prefixes within the refused space only; link-local,
// unspecified, multicast and the AWS IPv6 metadata address
// (fd00:ec2::254) are NEVER permitted, whatever the list says — the
// metadata ranges are what this guard exists among other things to
// protect.
//
// A pointer so "absent" (off) stays distinguishable from an explicit
// enabled: false, mirroring DeclineResponsesWebSockets. Listener.* field:
// changing it requires a restart, like every other listener knob.
EgressGuard *EgressGuardConfig `yaml:"egress_guard,omitempty" json:"egress_guard,omitempty"`
}

// EgressGuardConfig configures the forward proxy's SSRF dial guard.
type EgressGuardConfig struct {
// Enabled arms the guard. False (or an absent egress_guard block) keeps
// the dial behavior unchanged.
Enabled bool `yaml:"enabled" json:"enabled"`

// PermitCIDRs lists CIDR prefixes within the refused space that remain
// dialable — e.g. ["127.0.0.1/32"] for a laptop proxy whose upstream is a
// local model, or ["10.0.0.0/8"] for an in-cluster deployment. Never has
// any effect on link-local, unspecified, multicast or the AWS IPv6
// metadata address (fd00:ec2::254).
PermitCIDRs []string `yaml:"permit_cidrs,omitempty" json:"permit_cidrs,omitempty"`
}

// DeclinesResponsesWebSockets reports whether the forward proxy answers a WebSocket
Expand Down
34 changes: 34 additions & 0 deletions core/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1070,6 +1070,40 @@ func TestListenerConfig_DeclineResponsesWebSockets(t *testing.T) {
}
}

// TestListenerConfig_EgressGuard pins the default (off) and the block that arms
// it. Unset must mean off with or without a listener block: a sidecar's ordinary
// upstreams live in exactly the refused space, so a config predating the key
// turning the guard on would break the deployment it was written for.
func TestListenerConfig_EgressGuard(t *testing.T) {
var noBlock Config
if err := yaml.Unmarshal([]byte("mode: proxy-sidecar\n"), &noBlock); err != nil {
t.Fatal(err)
}
if noBlock.Listener.EgressGuard != nil {
t.Error("no listener block: EgressGuard is non-nil, want off")
}

var armed Config
if err := yaml.Unmarshal([]byte("listener:\n egress_guard:\n enabled: true\n permit_cidrs: [\"127.0.0.1/32\", \"10.0.0.0/8\"]\n"), &armed); err != nil {
t.Fatal(err)
}
g := armed.Listener.EgressGuard
if g == nil || !g.Enabled {
t.Fatalf("armed config: EgressGuard = %+v, want enabled", g)
}
if len(g.PermitCIDRs) != 2 || g.PermitCIDRs[0] != "127.0.0.1/32" || g.PermitCIDRs[1] != "10.0.0.0/8" {
t.Errorf("PermitCIDRs = %v, want the two configured prefixes", g.PermitCIDRs)
}

var presentButOff Config
if err := yaml.Unmarshal([]byte("listener:\n egress_guard:\n enabled: false\n"), &presentButOff); err != nil {
t.Fatal(err)
}
if g := presentButOff.Listener.EgressGuard; g == nil || g.Enabled {
t.Errorf("explicit false: EgressGuard = %+v, want present and disabled", g)
}
}

// TestLoad_PreAffinityLaptopConfigGetsClientAffinity loads a laptop config written before
// session.client_affinity existed — no session block at all, which is what every install
// made before it still has on disk. With affinity off, Bob Shell's header-less startup
Expand Down
9 changes: 8 additions & 1 deletion core/cost/usage/usage.go
Original file line number Diff line number Diff line change
Expand Up @@ -1270,8 +1270,15 @@ func (a *Aggregator) evictColdestLocked() {
// 426 is Cortex telling Codex to use HTTPS, which it then does, once per session: by
// design and on the default path, so counting it would give every healthy Codex
// fleet a nonzero error rate. The Finisher sees it as an allow for the same reason.
//
// And except the egress guard's refusal (pipeline.ErrorKindDestinationPrivate). Same
// shape, same reasoning: its 403 is the guard doing exactly what the operator
// configured it to, on traffic that was never going to be allowed through — a
// nonzero "error rate" made of refusals is the policy working, not the proxy
// failing. The Finisher sees it as an allow for the same reason.
func countsAsError(e *pipeline.SessionEvent) bool {
if e.Error != nil && e.Error.Kind == pipeline.ErrorKindUpgradeDeclined {
if e.Error != nil && (e.Error.Kind == pipeline.ErrorKindUpgradeDeclined ||
e.Error.Kind == pipeline.ErrorKindDestinationPrivate) {
return false
}
return e.StatusCode >= 400 || e.Phase == pipeline.SessionDenied
Expand Down
26 changes: 26 additions & 0 deletions core/cost/usage/usage_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -549,6 +549,32 @@ func TestRecord_DeclinedUpgradeIsNotAnError(t *testing.T) {
}
}

// Same shape for the egress guard: its 403 is the guard doing what the operator
// configured, so a refusal must not read as a failing proxy. The exemption is the
// kind, not the status — an origin's own 403 is still an error.
func TestRecord_DestinationPrivateIsNotAnError(t *testing.T) {
now := time.Date(2026, 9, 4, 23, 30, 30, 0, time.UTC)
a := New(WithClock(fixedClock(now)))

refused := respEvent(now, 403, 0, "", 0)
refused.Error = &pipeline.EventError{Kind: pipeline.ErrorKindDestinationPrivate}
a.Record("s1", refused)
origin403 := respEvent(now, 403, 0, "", 0)
origin403.Error = &pipeline.EventError{Kind: "backend_error", Code: "403"}
a.Record("s1", origin403)

b := a.Snapshot(time.Minute, BucketWidth, "", GroupStatus).Buckets[0]
if b.Requests != 2 {
t.Errorf("requests = %d, want 2", b.Requests)
}
if b.Errors != 1 {
t.Errorf("errors = %d, want 1 — the origin's 403 only, not the guard's refusal", b.Errors)
}
if _, ok := b.Series["403"]; !ok {
t.Errorf("both 403s still belong under their status; got keys %v", keys(b.Series))
}
}

// Request events carry no status, duration or usage. Counting them would double
// every request.
func TestRecord_IgnoresRequestPhase(t *testing.T) {
Expand Down
Loading
Loading