Repository navigation
feat: make a kape-dispatched sandbox actually reachable and credentialed (kape#274) - #283
Merged
Merged
Conversation
The controller launches kape-daemon as the sandbox main process (ADR-0086), but the pi-openshell base carries no kape binary, so the supervisor's spawn failed with 'No such file or directory (os error 2)' and the sandbox never reached Running (kape#274 AC1). Adds a golang build stage for ./daemon (standalone module, GOWORK=off) and installs the static binary at /usr/local/bin/kape-daemon — on the base PATH and under the policy's read_only /usr grant. BASE default moves to the released 0.3.0 pi-openshell tag, which carries the #273 policy fix.
Every sandbox the controller created with a credential binding reached READY and was then rolled back, so kape#274 could never get past AC1: apply network policy: UpdateConfig(policy merge): FailedPrecondition: network policies 'm1_credential_bindings' endpoint[0] (github.com:443) and 'github-ssh-over-https' endpoint[0] (github.com:443) overlap on port(s) 443 with conflicting metadata: credential_binding.provider="..." vs "" The additive merge (#264) was proven against m1_model_gateway, a rule the base policy does not declare. The sandbox image does declare the bound hosts — pi-openshell allows github.com:443 (github_ssh_over_https, pi) and api.github.com:443 (github_rest_api) — so appending a second declaration of the same endpoint with credential metadata makes the gateway's endpoint-ambiguity validator reject the whole merge. Bindings now merge against the sandbox's current policy (GetSandboxConfig): endpoints the policy already declares are stamped in place, keeping their own L7 restrictions (github stays read-only git smart-HTTP) and binary sets; hosts the policy does not declare keep the additive m1_credential_bindings shape. An endpoint already bound to a different provider fails closed rather than being silently overridden. Verified live on v0.1.0 that AddRule upserts by POLICY KEY, not by the rule's `name` field: a name-form key produced a duplicate rule instead of updating the existing one, so the ops are keyed by policy key with the body's name intact. Also logs sandbox rollbacks (agent.go): the delete error and its cause were dropped, which is why this failure surfaced nowhere except an opaque caller error.
The credential-stamping fix got sandbox creation green end to end (AC1), but the first prompt immediately failed: daemon rejected prompt: Post "http://<pod-ip>:9090/commands": context deadline exceeded The controller dials the daemon at the sandbox POD IP via the CR resolver (D7), while kape-daemon's compiled default (KAPE_DAEMON_ADDR) is 127.0.0.1:9090 — reachable only from inside the sandbox. The gateway already injects env for the main-process daemon; KAPE_DAEMON_ADDR now pins the bind to 0.0.0.0:<KAPE_DAEMON_PORT>, so the address the controller dials and the address the daemon listens on are the same port by construction.
…oxes NetworkCredentialBinding.provider is documented as "name of the provider attached to this sandbox whose static credentials may be resolved". The controller attached only the model provider, so a binding's provider was never present on the sandbox and the egress proxy had no credential to resolve — AC2 and AC3 of kape#274 would have failed as anonymous 401s no matter how correct the policy merge was. CreateSandbox now sends the model provider plus every provider named in the binding table, deduped, ordered by the table. Attachment and policy merge read the same table, so they cannot drift.
Two gaps made the pull transport unprovable end to end, both decided by the owner: 1. Policy merge (controller). The stamping merge read the policy and wrote it back unguarded. The gateway composes provider rules into the policy after create, so the merge raced that composition: the call returned OK and the credential metadata never reached the enforced policy — effective policy ended up with bare github endpoints, git/gh egressed anonymously, and the only visible symptom was a 401 inside the sandbox. The merge is now a bounded read-modify-write pinned to the sandbox resource_version (the token the gateway actually validates), retrying ABORTED and surfacing anything else. A merge that cannot converge is a loud failure, not a silent discard. 2. Daemon token (operator). The controller authenticates to the in-sandbox daemon with a bearer token (ADR-0083 D5) that nothing anywhere supplied: KAPE_DAEMON_TOKEN was absent from the CR, the Deployment env, and the sandbox env, so the daemon 401'd every request. The operator now generates one 32-byte hex token per KapeAgentServer, stores it in an owner-referenced Secret, and renders KAPE_DAEMON_TOKEN as a SecretKeyRef — the value never enters the Deployment spec, the CR, or CR status. Ensure is idempotent (a regenerated token would strand running sandboxes) and repairs an empty Secret rather than accepting it.
…oint The pull transport could not reach kape-daemon at all. ADR-0083 D7 dials status.podIPs, but OpenShell installs `openshell-sandbox-workloads`, whose only admitted ingress is the supervisor on :5500 — verified live: every non-supervisor packet to the workload pod is dropped, including hairpin traffic from the pod itself, and the controller's POST /commands died as a context deadline. Probes from a throwaway pod in the same namespace timed out identically. Reachability now uses the platform's supported path: - CreateSandbox is followed by ExposeService on the daemon port, so the gateway routes an endpoint through the supervisor's control plane to the daemon's LOOPBACK port inside the sandbox (which is why the supervisor, not the controller, is the peer the NetworkPolicy admits). - The resolver returns https://<exposed authority>; the transport dials the GATEWAY and lets the URL host ride as Host/SNI, because the exposed authority is a gateway routing name, not cluster DNS. The gateway's certificate carries *.openshell.localhost, so verification still holds. - Exposure failures roll the sandbox back like any other provisioning failure; teardown and rollback withdraw the endpoint (allow_missing). ExposeService is idempotent per sandbox, so a restarted controller re-derives the address instead of holding state.
Reaching the daemon through the gateway's exposed endpoint made the 0.0.0.0 bind unnecessary: ExposeService targets 127.0.0.1 inside the sandbox, and the workload NetworkPolicy denies every peer except the supervisor, so a wider bind buys nothing and only widens the sandbox's own attack surface. Reverts the bind to the daemon's compiled default while keeping KAPE_DAEMON_ADDR pinned to KAPE_DAEMON_PORT so the dial and bind ports cannot drift.
…on bearer token The exposed service endpoint defaults to STRIP, which removes the caller's Authorization header — precisely the ADR-0083 D5 daemon credential — so every pull failed 401 while the endpoint itself was reachable. BEARER_PASSTHROUGH is the only mode that lets the daemon authenticate. That field does not exist in v0.1.0: ServiceAuthorizationMode arrived in OpenShell v0.1.3 (verified against upstream tags). The vendored protos are therefore re-pinned v0.1.0 -> v0.1.5 (verbatim copy, buf generate; datamodel unchanged, options +3, sandbox +35, openshell +389 lines) and the controller builds against them unchanged. Requires the gateway to be >= v0.1.3. An older gateway drops the unknown field and keeps stripping, so the token never arrives — the failure mode is a 401, not a broken exposure, which is what this cluster showed before the bump.
OpenShell v0.1.5 rejects a credentialed endpoint that is L4-only: credentialed endpoint 'github.com:443' in rule 'pi' uses L4-only; configure L7 inspection or explicitly set allow_uninspected_credentials: true The pi rule lists github.com bare, so stamping a credential binding on it produced a policy the gateway refuses. Stamped copies now upgrade a bare L4 endpoint to REST with an allow-all L7 rule; endpoints that already declare L7 (protocol, access preset or explicit rules) are untouched, so no path restriction the policy author wrote is widened.
A sandbox that has just been created reports an EMPTY policy: the gateway
backfills spec.policy from the image on the first UpdateConfig ("backfilled
spec.policy from sandbox-discovered policy" — the supervisor's repeated
"Server returned no policy" is the same window). Stamping against that empty
read found none of the image's endpoints and fell back to the additive
m1_credential_bindings rule, which then collided with the image's own rules:
'm1_credential_bindings' endpoint[0] (github.com:443) and
'github-ssh-over-https' endpoint[0] (github.com:443) overlap on port(s) 443
with conflicting metadata: credential_binding.provider="..." vs ""
The merge is now two-phase: if the read policy carries no rules, spend one
merge on the model-gateway rule to trigger the backfill, re-read, and only then
stamp. A policy that is still empty after that fails closed rather than adding
a second declaration of the endpoints. Binding validation also moved ahead of
every gateway call, so a malformed binding can no longer leave the model-gateway
rule applied and the credentials unbound.
KAPE_PI_COMMAND empty selects kape-daemon's BUILT-IN fake pi. Nothing injected it, so every turn in a real deployment answered from the scripted fake — the event log showed a canned read_file plus "Working on it." and no tool could run (kape#274 AC2/AC3 could never be measured). The main-process daemon now receives KAPE_PI_COMMAND=pi, alongside the spool path and bind address it already gets. The fake stays available: it is what KAPE_PI_AUTOSTART=0 / the fake-pi smoke posture use deliberately.
pi's first model call never connected: the sandbox boundary's SSRF guard rejects
private addresses, and the model gateway is a ClusterIP service. The symptom is
the worst kind — the turn hangs with no error, because pi is blocked mid-call.
NetworkEndpoint.allowed_ips exists for exactly this ("when non-empty, the SSRF
internal-IP check is replaced by an allowlist check; domain must resolve to an IP
in this list"). The controller now resolves the configured model-gateway host and
allowlists its private IPv4 addresses as /32 entries:
- public targets get no allowlist (the guard only rejects internal addresses)
- loopback and link-local are never listed (the gateway blocks them regardless)
- an unresolvable host fails closed before any merge, so a sandbox is never
created against a policy it cannot use
KAPE_MODEL_GATEWAY_URL still decides the endpoint; the allowlist follows it.
… override The allowlist from the previous commit resolved to the wrong address. An in-cluster model gateway is typically an ExternalName -> headless service, so the controller's first lookup yields a service-network address while the SANDBOX connects to the backing pod: verified live, allowed_ips carried 10.152.183.70 while the real endpoints were 10.1.124.247/.219, and connections to those were blocked. Every model call therefore still hung. allowedIPsFor now follows the CNAME chain and collects the private IPv4 addresses of both hops, so the pod addresses are allowlisted alongside the service one. For a deployment whose backing pod IPs move, KAPE_MODEL_GATEWAY_ALLOWED_IPS pins the allowlist outright (comma-separated IPs/CIDRs, used verbatim, logged at startup) — resolution is a convenience, not a guarantee.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
kape#274 asked for proof that a controller-dispatched sandbox can authenticate to GitHub without a token in its env or filesystem. Getting to first contact required fixing eight defects between the controller, the sandbox image and OpenShell 0.1.5. This PR ships those fixes and records — honestly — what is still unproven.
AC1 now passes:
POST /agents → 201, sandboxReady, daemon alive, prompt accepted, turn runs with real pi.Fixes
7d6f126kape-devimage shipped nokape-daemon, so the supervisor's canonical-process spawn failed (No such file or directory)423eb5aoverlap … conflicting metadata)56ed18a→03820070.0.0.0, settled on loopback once exposure landed)bbbb745b9c9444bde76c8openshell-sandbox-workloadsadmits only the supervisor) →ExposeServiceca9669cAuthorizationMode=STRIPdeleted the daemon token →BEARER_PASSTHROUGH; protos re-pinned v0.1.0 → v0.1.56e791498edec1dUpdateConfigbackfills it → two-phase merge7557d77KAPE_PI_COMMANDunset selected the daemon's built-in fake pi — every turn answered from a scriptd443b88allowed_ips92f1b19KAPE_MODEL_GATEWAY_ALLOWED_IPSoverrideOperator change in the same commits: one operator-generated daemon bearer token per
KapeAgentServer, in an owner-referenced Secret, rendered as aSecretKeyRef.Images built and pushed from this branch:
kape-controller:0.1.7-274,kape-operator:0.1.4-274d,kape-dev:0.3.2(+pi-openshell:0.3.1-openrouter).What is NOT proven (do not read this PR as AC2–AC6)
Verified inside a controller-created sandbox, using
openshell sandbox exec(a registered process —kubectl execis unregistered and silently cannot resolve DNS):grep -c credential_bindingon the effective policy → 0) even though the controller logs the merge was submitted. Filed as Decision: credential-binding observability — merges that the gateway silently drops #280.models.json, so pi calls the image'shttps://api.openrouter.ai/api/v1, which the policy blocks → the turn hangs with no error. Filed as Decision: pull-transport parity — models.json patch, env injection, and fake-pi default #279.Design questions filed instead of guessed
#277 endpoint ownership (image policy vs provider composition) · #278 retire ADR-0083 D7 (exposure + passthrough) · #279 pull-transport parity · #280 credential-binding observability · #281 OpenShell lifecycle state is not gitops · #282
KapeAgentServersurface gaps.Tests
go test ./controller/— ok (53 s,TestLiveSmokeexcluded: it needs a live OpenRouter key).go vetclean,gofmtclean.golangci-lintcannot run in this sandbox (cgo preprocessing blocked); CI should cover it.